From Variety:
“The lions share of development funding has gone towards gaming for entertainment,” says Todd Richmond, Director at the Tech & Narrative Lab at Pardee RAND and member of IEEE. “[But] there has been a parallel effort on using games for training and education – so-called ‘serious games.’”
At the Mixed Reality Lab, where Richmond is also a director, his team began exploring user intent in a project known as ESP or Early Synthetic Prototyping. In ESP, Richmond and his team studied members of the military as they performed exercises in a virtual world, experiments that put players into a multiplayer match to search for specific target objects. The ultimate aim, for Richmond, is to accurately reflect humanity in technology – what he calls being an analog soul in a digital world.
“Current digital games typically log events during the game,” says Richmond. “The player clicked on this, they moved here, they swung their sword there. Game companies, who are in the business of making money by keeping people playing, evolved the game engines and applications to collect data that would help them understand what players were doing, and how they might keep them playing and develop new capabilities that would further expand the player engagement.”
“Entertainment games have one main goal: keep players playing. Thus a game designer has to create an application that engages and entertains,” he continues. “Commercial developers want to know what a player does so that they can design better and more engaging experiences. For serious game development, knowing what is insufficient.”
“For the Army, training is a significant investment in time and effort and is ongoing throughout a Soldier’s career. When a person is performing tasks, there typically is a ‘right’ and a ‘wrong’ way to perform said task. Current simulations and training are designed to understand what a person does, and sometimes how they do it, but not why. As a trainer, if I don’t understand why someone made a mistake (or got something right), remediation becomes formulaic and might not be effective – you can be treating a symptom instead of an underlying cause.”
Richmond’s ESP program uses an eye tracker to track where on the screen the user is looking at any given time. A webcam is trained on their face as they’re playing, which is also recorded and analyzed using machine learning and facial recognition. In some experiments, players wore an EEG electrode-covered cap, which would track when the user had seen and recognized the object they were looking for. The match is recorded and then analyzed using software that breaks down every action performed. Players appear on the screen with lines sprouting out from them, representing their next move, and the next one, and the next.
In return, the team was able to gather surprisingly specific information, ranging from “smile level” and “speech rate” to numerical values that are used to estimate the player’s emotion as angry, happy, or surprised, positive or negative.
The result is a virtual world of cause-and-effect. When a vehicle collides into a wall and rolls into the path of bullets, the team rewinds to unknot the events that occurred in the seconds and minutes beforehand which may have influenced events. If the players went left, should they have gone right instead? Did they double back early on when searching for their target object? Were they looking the wrong way or talking? Were they potentially angry or surprised? Were they smiling?
“Tacit knowledge often is what drives our behaviors. What we’re trying to glean is some of what is going on unconsciously, likely through biometrics, and in the future, through EEG and other brain sensing. If someone turned left instead of right, why did they make that choice?”
“We know the data isn’t particularly accurate at this point. But we also know that these types of computer vision systems and algorithms, particularly when combined with Machine Learning techniques, will become very accurate and reproducible. So part of our work was to think about how one would engage that information once it does become better.”