Science Scope

Video: Mind-reading tech reconstructs moving images others see

Posting in Science

For the first time ever, scientists used brain imaging to determine what moving images a person is seeing.


If only getting into someone's head were as easy as putting his brain inside a machine.

Oh, actually, it is that easy.

Scientists, for the first time ever, used brain imaging to determine what moving images a person is seeing. What enabled this breakthrough was a brain imaging technique called functional magnetic resonance imaging (fMRI), which tracks the fluctuations of the brain's blood oxygen levels.

The fMRI technique has long been used to help researchers see what static images a subject was looking at. However, it was always thought impossible to use the technology to reconstruct moving images.

The new development could someday lead researchers to visualize others' dreams and memories, and it could enable them to also reconstruct the human visual system on a computer.

How they did it

To conduct their experiment, the researchers, Jack Gallant of the University of California at Berkeley and Shinji Nishimoto of the Helen Wills Neuroscience Institute, laid inside an fMRI machine for hours -- while watching innumerable movie previews.

They analyzed their fMRI data to see how the brain lit up during each second of footage. They then scrutinized that data even further to determine what was happening on the neuronal level.

The next step was to collect 18 million video clips -- none of which were of the previews the researchers had watched.

With that library, they asked their computer model to use the fMRI patterns they had logged to guess what the viewer was seeing at the time. The video below shows on the left what the subject (or, in this case, researcher) was actually seeing and, on the right, what the computer thought they were seeing based on the YouTube video database.

"Usually you only get that kind of accuracy in physics, not neuroscience," Benjamin Singer, an fMRI researcher at Princeton University who was not involved with the study, told Technology Review. "It's a tour de force that brings together decades of work."

The study, which was published in the September 22 issue of Current Biology, could, decades from now, help us see the images inside the brains of people who cannot communicate, such as stroke victims and coma patients. It could also be used to develop a computer that people with cerebral palsy or paralysis could direct with their minds.

via: Technology Review, Popular Science

photo: screenshot

Laura Shin

Features Editor

Laura Shin has been published in The New York Times, The Wall Street Journal and The Los Angeles Times, and is currently a contributor at Forbes. Previously, she worked at Newsweek, the New York Times, Wall Street Journal and LearnVest. She holds degrees from Stanford University and Columbia University's Graduate School of Journalism. Follow her on Twitter. Disclosure