Groundbreaking AI Technology Uses People’s Eye Movements to Predict Their Decisions

A researcher from the University of Maryland’s Robert H. Smith School of Business and his co-authors have created an advanced artificial intelligence-based eye-tracking technology, capable of using people’s eye movements to predict the decisions they’ll make.

Michel Wedel, Distinguished University Professor and PepsiCo Chair in Consumer Science, worked with Moshe Unger of Tel Aviv University and AlexanderTuzhilin of New York University to develop an AI algorithm, which they dubbed RETINA, that takes mere seconds to predict people’s choice behaviors.

“This is something AI technology is very good at – using data to make predictions,” says Wedel. “We developed a deep-learning algorithm using eye movement data that allows us to predict what people are going to choose before they have actually made that decision.”

The researchers used data from an experiment where participants viewed a comparison website with rows and columns of products and their features. The data recorded which products and featured people looked at as they were making their decisions on what to click on.

The RETINA algorithm can predict people’s decisions to a “surprisingly accurate degree for the types of decisions studied in our paper,” says Wedel. “The fact that we were able to predict decisions accurately using just a small portion of that data – the first 5 seconds on participants looking at this – was very exciting.”

One of the unique things about the algorithm is that it uses raw eye movement data from each eye, Wedel says. “It’s a lot of data – several hundreds of thousands of data points, with millions of parameters – and we use it for both eyes separately.”

He explains that researchers typically synthesize eye movement data into aggregated chunks of information, which can miss some information and certain types of eye movements. With their advanced machine-learning method, Wedel and his colleagues use the eye movement data of both eyes without aggregation to quickly make predictions about people’s decisions.

“We took only a very small portion of the data at the very beginning of when people were looking at the products, in the first 5-10 seconds of viewing the screen – well before they’ve actually made any decisions.”

The algorithm could be applied in many settings by all types of companies. For example, a retailer like Walmart could use it to enhance the virtual shopping experiences they are developing in the metaverse, a shared, virtual online world. Many of the VR devices people will use to explore the metaverse will have built-in eye tracking to help better render the virtual environment. With this algorithm, Walmart could tailor the mix of products on display in their virtual store to what a person will likely choose, based on their initial eye movements.

“Even before people have made a choice, based on their eye movement, we can say it’s very likely that they’ll choose a certain product,” Wedel says. “With that knowledge, marketers could reinforce that choice or try to push another product instead.”

There are also many potential applications outside of marketing, he says. “Eye tracking is used in many other fields, including medicine, psychology and psychiatry, usability and design, arts, reading, finance, accounting – anything where people are making decisions based on some kind of visual assessment.

And eye-tracking technology will only become more ubiquitous.

Wedel says a lot of tech companies are exploring more ways to use eye movement data, with many of the biggest players, including Meta and Google, having recently acquired eye-tracking companies. With front-facing cameras, it is now possible to track people’s eye movements from any personal smartphone, tablet or computer. It’s just not as accurate yet as the advanced eye-tracking hardware researchers currently use, says Wedel, and there is still the big issue of privacy concerns – companies need to ask permission from users.

With privacy issues worked out, consumers could really benefit from eye-tracking as well, says Wedel.

“If algorithms like this are used as part of decision support systems, marketers can help you make better decisions and find better products that meet your needs,” he says. “Benefits to people might be even bigger in other areas where eye tracking is used, like medical applications.”

The researchers are already working with at least one company to apply the algorithm and extend their research to optimize decision-making.

“We think eye tracking will become available at very large scales,” says Wedel. “The processing of the eye movement data typically has been very laborious. With this algorithm, we side-step a lot of that, so there may be many applications that we haven’t even thought about.”

Wedel and his co-authors detail the algorithm in new research, “Predicting Consumer Choice From Raw Eye‑Movement Data Using the RETINA Deep Learning Architecture,” published in the journal Data Mining and Knowledge Discovery.