Wow. I knew that research into this sort of thing was ongoing, but had no idea it had come so far.
Scientists Reconstruct Brains’ Visions Into Digital Video In Historic Experiment
Scientists Reconstruct Brains' Visions Into Digital Video In Historic Experiment Scientists Reconstruct Brains' Visions Into Digital Video In Historic Experiment UC Berkeley scientists have developed a system to capture visual activity in human brains and reconstruct it as digital video clips. Eventually, this process will allow you to record and reconstruct your own dreams on a computer screen.
I just can't believe this is happening for real, but according to Professor Jack Gallant—UC Berkeley neuroscientist and coauthor of the research published today in the journal Current Biology—"this is a major leap toward reconstructing internal imagery. We are opening a window into the movies in our minds."
Indeed, it's mindblowing. I'm simultaneously excited and terrified. This is how it works:
They used three different subjects for the experiments—incidentally, they were part of the research team because it requires being inside a functional Magnetic Resonance Imaging system for hours at a time. The subjects were exposed to two different groups of Hollywood movie trailers as the fMRI system recorded the brain's blood flow through their brains' visual cortex.
The readings were fed into a computer program in which they were divided into three-dimensional pixels units called voxels (volumetric pixels). This process effectively decodes the brain signals generated by moving pictures, connecting the shape and motion information from the movies to specific brain actions. As the sessions progressed, the computer learned more and more about how the visual activity presented on the screen corresponded to the brain activity.
An 18-million-second picture palette
After recording this information, another group of clips was used to reconstruct the videos shown to the subjects. The computer analyzed 18 million seconds of random YouTube video, building a database of potential brain activity for each clip. From all these videos, the software picked the one hundred clips that caused a brain activity more similar to the ones the subject watched, combining them into one final movie. Although the resulting video is low resolution and blurry, it clearly matched the actual clips watched by the subjects.
Think about those 18 million seconds of random videos as a painter's color palette. A painter sees a red rose in real life and tries to reproduce the color using the different kinds of reds available in his palette, combining them to match what he's seeing. The software is the painter and the 18 million seconds of random video is its color palette. It analyzes how the brain reacts to certain stimuli, compares it to the brain reactions to the 18-million-second palette, and picks what more closely matches those brain reactions. Then it combines the clips into a new one that duplicates what the subject was seeing. Notice that the 18 million seconds of motion video are not what the subject is seeing. They are random bits used just to compose the brain image.
Given a big enough database of video material and enough computing power, the system would be able to re-create any images in your brain.
In this other video you can see how this process worked in the three experimental targets. On the top left square you can see the movie the subjects were watching while they were in the fMRI machine. Right below you can see the movie "extracted" from their brain activity. It shows that this technique gives consistent results independent of what's being watched—or who's watching. The three lines of clips next to the left column show the random movies that the computer program used to reconstruct the visual information.
Right now, the resulting quality is not good, but the potential is enormous. Lead research author—and one of the lab test bunnies—Shinji Nishimoto thinks this is the first step to tap directly into what our brain sees and imagines:
Our natural visual experience is like watching a movie. In order for this technology to have wide applicability, we must understand how the brain processes these dynamic visual experiences.
The brain recorders of the future
Imagine that. Capturing your visual memories, your dreams, the wild ramblings of your imagination into a video that you and others can watch with your own eyes.
This is the first time in history that we have been able to decode brain activity and reconstruct motion pictures in a computer screen. The path that this research opens boggles the mind. It reminds me of Brainstorm, the cult movie in which a group of scientists lead by Christopher Walken develops a machine capable of recording the five senses of a human being and then play them back into the brain itself.
This new development brings us closer to that goal which, I have no doubt, will happen at one point. Given the exponential increase in computing power and our understanding of human biology, I think this will arrive sooner than most mortals expect. Perhaps one day you would be able to go to sleep wearing a flexible band labeled Sony Dreamcam around your skull. [UC Berkeley]
And a side by side video of the original and interpreted image:
Interesting that faces are so much clearer than everything else.
"There are two novels that can change a bookish fourteen-year old's life: The Lord of the Rings and Atlas Shrugged. One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world. The other, of course, involves orcs." - John Rogers
Isn't this just recording the processing of visual information, since it works in realtime? That seems pretty limited, but if all you want to do is record images that someone is currently looking at, bully for them.
Lord of the Abyss wrote:Interesting that faces are so much clearer than everything else.
Not really - the human brain has quite a bit of circuitry/processing devoted just to noticing and decoding human faces. That's why it's so easy to see faces in random visual stimuli like clouds and smoke, your brain is pre-disposed to see faces with minimal information.
A life is like a garden. Perfect moments can be had, but not preserved, except in memory.Leonard Nimoy.
Now I did a job. I got nothing but trouble since I did it, not to mention more than a few unkind words as regard to my character so let me make this abundantly clear. I do the job. And then I get paid.- Malcolm Reynolds, Captain of Serenity, which sums up my feelings regarding the lawsuit discussed here.
If a free society cannot help the many who are poor, it cannot save the few who are rich. - John F. Kennedy
Although this is really cool, it isn't a direct representation of what the brain is seeing. My understanding of what they did was look at brain activity while viewing a video clip and compare it to the brain activity from viewing millions of other random video clips. Then they took the video clips with similar brain activity to the one that the subject was viewing and spliced them together.
So I guess it is like having a subject look at a piece of fruit and matching their brain activity with a bunch of pictures of similar looking fruit. I look forward to seeing how this technology develops.
All this brain-scanning stuff just comes across to me as being incredibly sinister. As a scientist, I'd rather destroy it all and forgo all scientific progress in this area if it makes it even a little less likely that the government will get their hands on it.
fuzzymillipede wrote:Although this is really cool, it isn't a direct representation of what the brain is seeing. My understanding of what they did was look at brain activity while viewing a video clip and compare it to the brain activity from viewing millions of other random video clips. Then they took the video clips with similar brain activity to the one that the subject was viewing and spliced them together.
That would explain why it's not just fuzzy, but there are actually details present that are clearly wrong. Still, with a large enough bank of comparisons, etc. it could in principle become arbitrarily accurate.
Yes, I can see the broad applications for mind-reading that a device like this would allow for. If only it didn't require the subject to sit in an MRI for hours at a time and the images weren't limited to videos shown to said subject. Definitely something they'll be deploying on the street to spy on your porn surfing habits in the near future
As for potential uses in the real world, it could easily be used to help us understand dreams (like they said) or maybe people suffering from locked-in syndrome.
At present - and assuming they don't arrest you or simply pass a law to require attendance. And what Western countries might do is one thing, what more dubious countries might do is quite another. Can you imagine, say, a brain scanning device that can tell if people are gay being deployed in Iran?
What, you mean like they can do right now by showing people porn and measuring their responses? No need for an MRI or prototype technology to figure out if someone gets aroused by stuff.
Although I sense we just went waay off topic, replacing that rather dubious metric with some sort of highly accurate diagnostic that can be applied even very young would indeed have consequences.
While Conqueror goes off the deep end into crazy town I have to say as a security specialist this is one of the big all time nightmares except for the fact it's not in it's current incarnation. If it get's good enough that you can use the images to read a piece of paper someone's looking at without looking at it when one starts to get antsy about where this technology is going to go. But at the present this is on the line of some guy making a paper airplane and running out to protest Nuclear Bombers.
Sure one day we might end up there, but short of giant MRI machines or surgery to implant a chunk of metal in your brain to record things which at bests show a very blurry image ehh... color me unconcerned. It's interesting but there's nothing there yet. I'd be interested when the use this technology with a much larger database (Like specifically gathered photos of ten million people and a century worth of stock video, IE a hundred years worth of stock video of the everyday (IE a few hundred trillions second's worth)
"A cult is a religion with no political power." -Tom Wolfe Pardon me for sounding like a dick, but I'm playing the tiniest violin in the world right now-Dalton
Maybe I'm tired, or maybe that article is poorly written, but I am a bit confused as to the purpose of the whole YouTube random video element of the experiment. This seems more to be an interesting use of computer and MRI technology than it is any sort of profound discovery related to human visual perception.
EDIT: What I mean to say is, color me impressed when the computer artificially creates a video clip based on only the raw MRI data. While this experiment is cool and all, what does it tell us? Besides that computer processing has improved to the point that it can reliably search a large video database? Nothing new about higher visual processing is learned.
Mr Bean wrote:While Conqueror goes off the deep end into crazy town I have to say as a security specialist this is one of the big all time nightmares except for the fact it's not in it's current incarnation. If it get's good enough that you can use the images to read a piece of paper someone's looking at without looking at it when one starts to get antsy about where this technology is going to go. But at the present this is on the line of some guy making a paper airplane and running out to protest Nuclear Bombers.
Sure one day we might end up there, but short of giant MRI machines or surgery to implant a chunk of metal in your brain to record things which at bests show a very blurry image ehh... color me unconcerned. It's interesting but there's nothing there yet. I'd be interested when the use this technology with a much larger database (Like specifically gathered photos of ten million people and a century worth of stock video, IE a hundred years worth of stock video of the everyday (IE a few hundred trillions second's worth)
Of course in its present incarnation it's not much to write home about. But the future implications are the whole point. The technology will get better, and it will become miniaturised and cheap. It may not happen in our lifetimes - though I wouldn't necessarily bet against that - but eventually this sort of technology is going to give governments enormous potential power, which may be impossible to subsequently overturn.
Governments already have enormous power, autocratic government have been able to make people vanish for thousands of years as a tool of control. Being able to see what you see is quite different from being able to see what you've seen. And that's not something this technology is trending towards. These experiments have nothing to do with memory but only visual input.
"A cult is a religion with no political power." -Tom Wolfe Pardon me for sounding like a dick, but I'm playing the tiniest violin in the world right now-Dalton
HMS Conqueror wrote:Although I sense we just went waay off topic, replacing that rather dubious metric with some sort of highly accurate diagnostic that can be applied even very young would indeed have consequences.
Seeing a file applied to the teeth or pliers applied to nails would produce better results faster than even very advanced version of this tech, I quite fail to see the concern here. When was last time you heard of FBI doing that?
In hundred years, maybe, but the law and countermeasures to this will likely evolve faster than that. Who knows, maybe we'll see the day when tinfoil hat will be genuinely useful item?
Not really. Torture is extremely unreliable. The point is to eliminate the people who actually hold those views/have those characteristics (whatever the state thinks is undesirable), not to brutalise the population at random.
The law is evolving to lessen existing protections, not create new ones.
query - it's currently run on the brain reaction to visual input.
does imagining an image in your head also activate visual processing neurons?
of course, you wouldn't actually be able to see anything the subject couldn't imagine clearly enough to draw - and I imagine writing is processed elsewhere (although a database of scans from people reading could have the same effect, assuming no dyslexics or other outliers)
"Aid, trade, green technology and peace." - Hans Rosling.
"Welcome to SDN, where we can't see the forest because walking into trees repeatedly feels good, bro." - Mr Coffee
I'm thinking of the creative potential here. If this divice can scan things we imagine the possibilities for creativity are endless.
"The real ideological schism in America is not Republican vs Democrat; it is North vs South, Urban vs Rural, and it has been since the 19th century."
-Mike Wong
yes, but the ways you can apply paint to canvas are endless too - but only a tiny fraction of the pictures could be said to have artistic merit.
"Aid, trade, green technology and peace." - Hans Rosling.
"Welcome to SDN, where we can't see the forest because walking into trees repeatedly feels good, bro." - Mr Coffee
madd0ct0r wrote:yes, but the ways you can apply paint to canvas are endless too - but only a tiny fraction of the pictures could be said to have artistic merit.
Which is why digital art will never, ever start to be seen as an art form... oh wait.
Gaian Paradigm: Because not all fantasy has to be childish crap. Ephemeral Pie: Because not all role-playing has to be shallow. My art: Because not all DA users are talentless emo twits. "Phant, quit abusing the He-Wench before he turns you into a caged bitch at a Ren Fair and lets the tourists toss half munched turkey legs at your backside." -Mr. Coffee
the possibilities for creativity with this method might be endless, but they already are with other mediums (including digital art, photography, poetry music ect)
The artistic merit of what is produced is as infinitely variable as what can be produced.
But we're aiming for the good stuff, no-matter what medium.
"Aid, trade, green technology and peace." - Hans Rosling.
"Welcome to SDN, where we can't see the forest because walking into trees repeatedly feels good, bro." - Mr Coffee
They are endless to people who have a sufficient level of skill to visualize their ideas in a certain medium.
I bet there are some superduperuber creative people out there who have no idea how to draw/paint/whatever and therefor their creativity is useless because they can´t share their visions.
This would remove the need to learn the craftsmanship aspect of visualizing an idea.
salm wrote:This would remove the need to learn the craftsmanship aspect of visualizing an idea.
At least if it worked that way.
It doesn't; when you visualise something you project onto only the upper layers of the visual processing chain, I really doubt that's enough to extract anything useful. However it should be possible to develop an interface that detects the transformation you are trying to apply to the image you are starting at, and then does that in your paint program, e.g. if you imagine a part of the picture turning blue it does that. I am dubious whether a one-way stochastic scanning mechanism could ever provide enough precision to be more useful than a conventional interface though; the training time required is also likely to be at least as long as learning to use conventional tools.
That said there's no hard physical blocker to creating a magic 'turn my vague ideas into polished art' machine, but it would require a detailed live model of the source human's brain and pretty sophisticated AI, vastly harder than the already tricky task of recording the visual image stream.
I'm a little surprised no one has mentioned the movie Brainstorm, a sci-fi classic about full-sensory recording.
