Fighting for nonverbal communication

This is the first use of Voice4PIMD technology, a small collection of cameras, sensors, and speakers meant to capture nonverbal communication for people with profound intellectual and multiple disabilities (PIMD). Over time, the system will use artificial intelligence and machine learning, alongside user feedback, to translate an individual's facial expressions, gestures, hand motions, eye gazes, and more into text or audio commands for caregivers.

Fennell, an 83-year-old grandfather, is an unlikely inaugural test case for the innovative technology—but he is inspired, as always, by his granddaughter, Haley Hilt, sitting at his right. Her gaze is in her lap, but her head is cocked, a sign that she is listening to all that is unfolding around her, he explains. Hilt is a 19-year-old with CDKL5 deficiency disorder, a rare neurodevelopmental disorder characterized by frequent seizures and impairment of motor, cognitive, speech, and visual abilities. Hilt does not walk. She does not speak. And yet, while she may not use words, Fennell assures people that she says plenty.

Fennell knows when she is tired, when she is upset, when she doesn't care for someone, when she is teasing him. He firmly believes Hilt has thoughts, feelings, and opinions just like any other young adult.

"I just interpret body language, the smile, the body posture, the gestures, the vocalizations," he says. "After a while, they all mean something."

But the trick is decoding what each gesture, each utterance, each movement means. That requires an attentive eye—like that of her parents, her sister, her grandmother, and a few trusted aides—or, perhaps, new technology like the box in front of them.

From diagnosis to nonverbal communication advocacy

At 4 weeks old, Hilt had her first seizure. Then another. And another. Doctors first thought perhaps it was a normal infant startle reflex. But her mother, Melissa Hilt, a nurse, suspected otherwise. She bought a camcorder and taped the spasms for doctors to watch. She was correct—at just a few months old, Haley had had dozens of seizures, with no explanation as to why.

It took two and a half years for Haley's official diagnosis—a variant in the CDKL5 gene that produces a protein critical for normal brain development. Even once they understood the cause, there was little information on the disorder. Haley was one of the first 100 people with the diagnosis. As Fennell recalls, doctors implied it would eliminate all cognition and would all but erase the future they had envisioned for her. Fennell and Melissa cried in the lobby of the hospital. But once the grief subsided, Fennell, a former social worker, realized he disagreed with the assessment.

Hilt was diagnosed at just two years old, but information about CDKL5 was sparse.

"I said, 'No! I've helped raise that girl since she was only a few months old, as soon as my daughter went back to work. I've had her with me every day—eight, nine, 10 hours a day. I know that child's got an intellect!"

And so began a devoted grandfather's advocacy.

To start, Fennell printed pictures. One of Mom. One of Dad. Grandma. Grandpa. Cat. Dog. He'd place one photo at either end of a 4-foot-long coffee table, and balance Haley on his knees in the middle.

"I would put out the pictures and say, 'Which one is Mom?' Bang! The head would go right to Mom. I did that over and over and over again over a few days. The accuracy was stunning."

"What did I prove? . . . She could see. She could hear. She had a vocabulary, because she was responding."

In short, 2-year-old Haley had more intellect than experts believed she did.

Today, Haley claps when she's excited—like while there's conversation about her dogs, Shadow and Boots, or about her vacation spent floating on a makeshift raft on a lake. When asked to give up her "chewie," a comfort item, she grips it tight, resulting in a comical tug-of-war with her grandfather. She watches her grandfather turn the pages of a favorite book. She leans to the side when she wants to lay down. She hums a quiet "mmmm" and grasps her grandfather's hand while nervous. She has opinions, and she does communicate them. It just requires someone, or something, to vocalize them for her.

She hums a quiet "mmmm" and grasps her grandfather's hand while nervous.

"The vast majority of nonverbal or minimally verbal people do have something to say. They're people! They have joy. They have sorrow. They have pain. They have great feelings. They just don't have the language to do it," Fennell says.

To better illustrate the point, Fennell remarks that nonverbal communication is everyone's first language. He highlights how babies have a hungry cry, an angry cry, a tired cry. They rub their eyes. They recognize faces and voices. They respond to human touch, to textures. They respond to their name. Why couldn't the same types of communication be true for adults without speech?

Fennell started looking for people who believed him, and who could work with him to take seriously the gravity behind nonverbal communication. He started researching. And speaking. And advocating. Attending conferences. Eventually, it led him to Michal Kosiedowski and the Poznan Supercomputing and Networking Center in Poland, who, with some funding from the European Union, had started looking at interpreting nonverbal behavior for individuals with profound intellectual and multiple disabilities.

That is where he met Jerek Nabrzyski.

AI and augmented communication