How to programmatically get Head Gaze position and direction in MRTK3 for Hololens2

Question

I working with a Microsoft Hololens2 and the Windows Mixed Reality Toolkit (MRTK3). I'm trying to programmatically get hold of the current Gaze direction and origin, which is explained here for MRTK2:

void LogGazeDirectionOrigin()
{
    Debug.Log("Gaze is looking in direction: "
        + CoreServices.InputSystem.GazeProvider.GazeDirection);

    Debug.Log("Gaze origin is: "
        + CoreServices.InputSystem.GazeProvider.GazeOrigin);
}

However, how do I do this in MRTK3?

Searched on Windows forums with no luck

Answer 1

For future reference: To get gaze direction in MRTK3 you have multiple options

1. FuzzyGazeInteractor: By default, the MRTK XR Rig has the MRTK Gaze Controller attached as a child which contains the FuzzyGazeInteractor. For more information see this documentation by Microsoft. To get gaze origin and direction, you can then do the following, assuming gazeInteractor is a reference to the FuzzyGazeInteractor:

   Vector3 gazeOrigin = gazeInteractor.rayOriginTransform.position;
   Vector3 gazeDirection = gazeInteractor.rayOriginTransform.forward;
   

2. ExtendedEyeGazeDataProvider: In contrast to regular eye tracking, this interface allows extraction of additional, more detailed information such as separated gaze information about left & right eye. See this documentation by Microsoft in that regard. Getting gaze direction & origin (for example for combined gaze) then works as follows, assuming gazeProvider is a reference to the ExtendedEyeGazeDataProvider:

    var combinedGazeReadingInWorldSpace = extendedEyeGazeDataProvider.GetWorldSpaceGazeReading(ExtendedEyeGazeDataProvider.GazeType.Combined);
    Vector3 gazeOrigin = combinedGazeReadingInWorldSpace.EyePosition;
    Vector3 gazeDirection= combinedGazeReadingInWorldSpace.GazeDirection;

3. Regarding head gaze: If you are actually interested in the user's head gaze, you can just use information about the MainCamera. Head gaze origin & direction could then be estimated using the following approach:

    Vector3 headGazeOrigin = Camera.main.transform.position;
    Vector3 headGazeDirection = Camera.main.transform.forward;

Also, if no eye tracking is available, the named interfaces will return head gaze as a fallback. You could also disable eye gaze composite completely so that head gaze is always used (see this github issue as an example).