Human Computer Interaction

Robust Pupil Detection and Gaze Estimation

The reliable estimation of the pupil position in eye images is perhaps the most important prerequisite in gaze-based HMI applications. While there are many approaches that enable accurate pupil tracking under laboratory conditions, tracking the pupil in real-world images is highly challenging due to changes in illumination, reflections on glasses or on the eyeball, off-axis camera position, contact lenses, and many more.

Description MAM

Description CBF

Description BORE

Description PUREST

Description PURE

Description ElSe

The input to the algorithm is a grayscale image. After normalization, a Canny edge filter is applied to the image. In the next algorithmic step, edge connections that could impair the surrounding edge of the pupil are removed. Afterwards connected edges are collected and evaluated based on straightness, intensity value, elliptic properties, the possibility to fit an ellipse to it, and a pupil plausibility check. If a valid ellipse describing the pupil is found, it is returned as the result. In case no ellipse is found a second analysis is conducted. To speed up the convolution with the surface difference and mean filter the image is downscaled. This operation is performed by calculating a histogram for all pixels from the large image influencing the pixel in the downscaled image. In each histogram, the mean of all intensity values up to the mean of the histogram is calculated and used as a value for the pixel in the downscaled image. After applying the surface difference and mean filter to the rescaled image, the best position is selected by multiplying the result of both filters and selecting the maximum position. Choosing a pixel position in the downscaled image leads to a distance error of the pupil center in the full scale image. Therefore, the position has to be optimized on the full scale image based on an analysis of the surrounding pixels of the chosen position.

Description ExCuSe

The input to the algorithm is a grayscale image. In the first step the color histogram is analysed and the algorithm decides if he expects the pupil to be bright or dark. For bright pupils the algorithm selects the curvedd edge with the darkest enclosed intesity value. This is done by refining the result of the canny edge filtered image with morphologic operations. The remaining edges are analysed if they are curved or straight. For the curved edges the eclosed intesity value is calculated and the best is selected. If the algorithm expects the pupil to be dark a threshold based on the standard deviation is calculated. The image gets thresholded and a coarse positioning is performed similarly to the radon transformation. This coarse position is optimized by moving to near intesity values which are darker or equal dark to their neigbourhood. After the optimization the image is threasholded by an increased threashold. All edges which are to far from pixels under the threashold are removed. The remaining edges get filterd as described befor for the bright pupil. Starting from the optmized position the surrounding edges are used to fit an ellipse.


Results remote (Proposed = BORE)

Results microscope


  • ExCuSe C++ code
  • ExCuSe Matlab code
  • ElSe C++ code (morphologic split)
  • ElSe C++ code (algorithmic split)
  • ElSe C++ code (morphologic split adjustable validity threshold for blink rejection)
  • ElSe C++ code (algorithmic split adjustable validity threshold for blink rejection)
  • ElSe C++ code (remote configuration)
  • Reimplementation George [Link]
  • Reimplementation Timm [Link]
  • Reimplementation Droege [Link]
  • Pupil detection through a ocular from a microscope
  • CBF C++ lib
  • BORE C++ lib
  • MAM C++ lib

All algorithms (ftp)

Data sets

  • ExCuSe data set
  • ElSe data set
  • Remote data set
  • Microscope pupil images

All algorithms (ftp)

Example results

Explanatory video (ExCuSe)


ElSe: Ellipse Selection for Robust Pupil Detection in Real-World Environments

by W. Fuhl, T. Santini, T. C. Kübler, and E. Kasneci

In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications (ETRA), pages 123–130, 2016.


Learn More

ExCuSe: Robust Pupil Detection in Real-World Scenarios

by W. Fuhl, T. C. Kübler, K. Sippel, W. Rosenstiel, and E. Kasneci

In 16th International Conference on Computer Analysis of Images and Patterns (CAIP 2015), 2015.


Learn More

Evaluation of State-of-the-Art Pupil Detection Algorithms on Remote Eye Images

by W. Fuhl, D. Geisler, T. Santini, and E. Kasneci

In ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct publication – PETMEI 2016, 2016.


Learn More

EyeLad: Remote Eye Tracking Image Labeling Tool

by W. Fuhl, T. Santini, D. Geisler, T. C. Kübler, and E. Kasneci

In 12th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), 2017.


Learn More

Non-Intrusive Practitioner Pupil Detection for Unmodified Microscope Oculars

by W. Fuhl, T. Santini, C. Reichert, D. Claus, A. Herkommer, H. Bahmani, K. Rifai, S. Wahl, and E. Kasneci

In Elsevier Computers in Biology and Medicine 79: 36-44, 2016.


Learn More