Search FQXi


Steve Dufourny: "There is no proof that the universe can be explained like this like an..." in Schrödinger’s A.I....

Georgina Woodward: "'verisimilitude' is a nice word." in Can We Feel What It’s...

R.H. Joseph: ""[H]ow consciousness plays with quantum mechanics, our theory of the very..." in Can We Feel What It’s...

Jim Snowdon: "If Earth rotated once a year, rather than once every 24 hours, every..." in The Nature of Time

Jim Snowdon: "The constant rotational speed of San Diego is 1,408 kilometers per hour. ..." in The Nature of Time

Steve Dufourny: "If we don t change the global system, we shall add an ocean of chaotical..." in Global Collaboration

Steve Dufourny: "Hi all, The world must absolutelly find adapted solutions , here is the..." in Global Collaboration

Himalya Quartz: "Here at our store, you can find a wide variety of healing and Reiki..." in Anatomy of spacetime and...

click titles to read articles

Schrödinger’s A.I. Could Test the Foundations of Reality
Physicists lay out blueprints for running a 'Wigner's Friend' experiment using an artificial intelligence, built on a quantum computer, as an 'observer.'

Expanding the Mind (Literally): Q&A with Karim Jerbi and Jordan O'Byrne
Using a brain-computer interface to create a consciousness 'add-on' to help test Integrated Information Theory.

Quanthoven's Fifth
A quantum computer composes chart-topping music, programmed by physicists striving to understand consciousness.

The Math of Consciousness: Q&A with Kobi Kremnitzer
A meditating mathematician is developing a theory of conscious experience to help understand the boundary between the quantum and classical world.

Can We Feel What It’s Like to Be Quantum?
Underground experiments in the heart of the Italian mountains are testing the links between consciousness and collapse theories of quantum physics.

October 1, 2022

Seeing is Believing
Neuroscientists are testing the idea that what we see when we perceive the world is ultimately an internal projection.
by Miriam Frankel
FQXi Awardees: Doris Tsao, Janis Hesse
March 22, 2022
Bookmark and Share

Credit: GrAI, Shutterstock
You are walking across a vast grass field on a misty morning when you suddenly spot a lake in the distance. As you get closer, you notice a lone duck swimming peacefully across it. But this image doesn’t last. After walking another ten yards, you realise your mistake—it isn’t a duck but a plastic bottle. Your perfectly smooth conscious image changes rather abruptly, flipping seamlessly from duck to bottle.

Conscious visual perception—what we are conscious of seeing—is typically explained by light bouncing off the various items around us and hitting our eye’s retina. The signal then travels to the back of our brain for visual processing and is fed forward through more complex areas until the brain can generate a perfectly coherent conscious image. A similar process takes place for other types of perception. But this raises a puzzle: if true, why is our conscious perception seemingly so perfect, lacking gaps even while we struggle to discern something?

This is one of many questions that has led some researchers to suspect that our conscious perception isn’t actually down to our brains processing what we experience with our senses. Instead, they suggest, we experience with our senses what our brain believes there to be; our brain helpfully fills in any gaps in perception. It is incredibly hard to prove this idea that what we see when we perceive the world is ultimately an internal projection. But Doris Tsao and Janis Hesse, neuroscientists at Caltech, in Pasadena, California, are tackling the problem with a series of experiments, aided by a $100,000 grant from FQXi.

Tsao’s interest in consciousness started at college, when she read Julian Jaynes’ 1976 book, The Origin of Consciousness in the Breakdown of the Bicameral Mind. "It was the first time I became curious about anything," she explains. "The cause of consciousness just seemed like the only truly interesting question in science."

Hesse has a similar attitude: "Consciousness is the most familiar thing to you—being the first thing that is with you when you wake up and the last thing to leave when you fall asleep—but at the same time its relation to the physical world and the brain is the biggest mystery in science," he says. "It is almost magical in that sense."

Obama or Taco?

The duo is using a particular experimental technique to probe what happens in the brain when we suddenly become conscious of something. This technique, which is called ’binocular rivarly,’ involves placing two distinctly different images—such as a picture of Barack Obama and a picture of a taco—in front of each eye respectively. You can try this now and see that what you become conscious of isn’t a mix of the two images. Rather, we perceive only one image at a time—either the taco or Obama—while being unconscious of the other. It is similar to the famous face-vase illusion, in which one can either see a vase or two faces—but not both at the same time (image above, right). What changes when we flip from one image to the other is therefore our conscious perception—our visual input remains exactly the same.

Doris Tsao
To investigate what happens in the brain during such changes in consciousness, the team is working with macaque monkeys, which have brains similar to those of humans. Specifically, they are zoning in on a network in the brain, previously discovered by Tsao, called the face-patch system. This consists of six distinct areas, most of which are located in two finger-shaped regions known as "the inferior temporal cortex" (IT cortex) towards the back of the brain. By inserting tiny, sensitive electrodes into these areas of the macaque brain, they can record the activity of thousands of individual neurons.

In 2017, this technique helped Tsao and her colleague Steven Le Chang to crack the exact code that these cells use to decipher a face (Chang, L. & Tsao, D. Y. Cell 169(6):1013-1028.e14 (2017)). While showing the monkeys images of faces, they discovered that cells in different patches code for specific things, such as some aspect of dimension or appearance. A specific cell can also code for several different things, such as distance between the eyes and length of the nose, with a unique firing pattern in each case. Amazingly, the team could even reconstruct an image based purely on a neuronal pattern.

It is because neuroscientists now know the face patch system so well that Tsao and Hesse are choosing to do their current research on faces. The face patches form a hierarchy in which lower level patches code for faces at specific views—more camera-like representation—while higher level patches code for identity of faces, invariant to view. "It’s a mini hierarchy of the brain," explains Tsao, adding that there are other patches coding for other objects in a similar way.

The approach has already yielded some results. Neuroscientists know that the cells in these patches fire differently depending on whether a monkey is looking at a picture of Obama or a picture of a taco. What Tsao and Hesse were interested in, however, was what would happen during binocular rivalry. Would the team see an instant swap from the highly active Obama brain pattern to the more silent taco pattern? If so, that would be a direct neural correlate of that conscious perception.

But nothing is ever simple when it comes to the brain. Rather than seeing a switch between two distinct patterns, they found that the cells encoded for both images simultaneously (preprint: bioRxiv 2020.04.22.047522 (2020)). "Previously people in the field believed that these face cells in the IT cortex basically represent your consciousness—that what they encode is exactly what you are conscious of," explains Hesse. But, assuming that the monkeys, like humans, are conscious of only one image at a time, "this data leaves open the question of how consciousness really is represented in the brain," Hesse says.

Mechanics of Consciousness

The duo is now digging deeper. Rather than just looking at how consciousness is represented in the brain, they want to discover its underlying mechanics. In their next experiment, they therefore plan to record cells from different areas in the hierarchy. While there are bottom-up pathways from simpler to more complex areas, there are also "feedback" connections going the other way. "We don’t really know what the feedback is for," says Hesse. "But there are as many backwards connections as forwards connections in the brain."

Janis Hesse
Recording with primate neuropixels.
Credit: Tsao and Hesse
Their hypothesis is that conscious perception arises entirely through top-down signals rather than the other way around. "The brain has this high level model of the world which will feed back to predict and recreate the input in lower order sensory regions—it’s kind of like a computer graphics engine," explains Hesse. "So what we are perceiving is not the brain processing input but what the brain believes there to be, shaped by our prior experience and beliefs."

Tsao thinks this would explain why vision is so strangely coherent. "We think all of vision is due to this, whether you are perceiving a visual stimulus or visualising in the mind’s eye." In fact, while the top-down process that generates our conscious awareness may often be triggered by bottom-up signals from our senses—updating the brain’s model of reality by telling it there is a mismatch between the model and the visual system—this doesn’t seem to be necessary for conscious perception. Experiments dating back decades have already shown that you can create vivid hallucinations (both visual and other), a type of conscious experience, by electrically stimulating a person’s temporal lobe, which contains the IT cortex. This suggests the bottom-up signals from our senses aren’t necessary for conscious awareness, but may just help to continually tweak our conscious perception.

Excitingly, a recent independent study in mice, by neuroscientists in Germany, seems to support this hypothesis. In the brain’s most basic cells, called pyramidal cells, specific ’dendrites’—extensions that receive communication from other cells—get forward input while others get feedback. The study showed that when mice become unconscious due to anaesthesia, it is the dendrites that receive the feedback that are affected, suggesting there is a deep connection between top-down processing and consciousness at the cellular level.

is almost magical
in that sense.
- Janis Hesse
Hesse and Tsao don’t just want to show that feedback is at work when we become conscious of something new. They also want to track the exact chain: where does the feedback start and where does it end? Does conscious perception require any involvement from the prefrontal cortex, which is normally involved in cognitively demanding tasks, or is it somehow generated in the IT cortex or elsewhere, from which it feeds back to simpler visual processing areas?

Matthias Michel, a philosopher at the Center for Mind, Brain, and Consciousness at New York University, New York, says it is a "good approach," but warns that you have to work very hard to make sure that the difference in signal you get during binocular rivalry is purely down to alternation in consciousness, rather than other confounding factors, such as thinking about an item. "Tsao is really an expert on face processing, so if anyone can do it correctly it is her," he adds.

Christof Koch, a neuroscientist at the Allen Institute for Brain Science, in Seattle, Washington, is also impressed. "It is a beautiful experiment," he says. Koch is hoping the approach can one day give some deep insights into how to best approach consciousness in a scientific way. "It speaks to the debate about whether consciousness is created within, in the back of the brain, or whether it requires critical input from the front of the brain."

Comment on this Article

Please read the important Introduction that governs your participation in this community. Inappropriate language will not be tolerated and posts containing such language will be deleted. Otherwise, this is a free speech Forum and all are welcome!
  • Please enter the text of your post, then click the "Submit New Post" button below. You may also optionally add file attachments below before submitting your edits.

  • HTML tags are not permitted in posts, and will automatically be stripped out. Links to other web sites are permitted. For instructions on how to add links, please read the link help page.

  • You may use superscript (10100) and subscript (A2) using [sup]...[/sup] and [sub]...[/sub] tags.

  • You may use bold (important) and italics (emphasize) using [b]...[/b] and [i]...[/i] tags.

  • You may also include LateX equations into your post.

Insert LaTeX Equation [hide]

LaTeX equations may be displayed in FQXi Forum posts by including them within [equation]...[/equation] tags. You may type your equation directly into your post, or use the LaTeX Equation Preview feature below to see how your equation will render (this is recommended).

For more help on LaTeX, please see the LaTeX Project Home Page.

LaTeX Equation Preview

preview equation
clear equation
insert equation into post at cursor

Your name: (optional)

Recent Comments

Hello Mr Schneider, I ahve always loved and been fascinated by the magicians, when I was younger , a child I was more than fascinated, I thought it was real when I was a child lol .I am understanding what you tell us about this subconscious and this consciousness , best regards

I am a magician. Perhaps you may find some of my thoughts interesting. I am capable of creating reality in the human mind. I have detailed this information in a book titled, The Theory and Practice of Magic Deception.

My conclusion is the mind gathers data from many sources. This data is fed to an internal projection room. The ‘I’ is then an observer in this room and consciously observes the 3D videos projected. The subconscious (I don’t have a better word for this) constructs these...

Interesting concept, and surely with some basis to be at least partly true. However, why don't we keep hallucinating expectations after we close our eyes? That would be a very critical time of need! I wrote a piece dealing with issues of conscious experience for the FQXi Essay Contest "How Should Humanity Steer the Future?" (2014.)

(Pardon not getting LaTeX URL to work.)

read all article comments

Please enter your e-mail address:
Note: Joining the FQXi mailing list does not give you a login account or constitute membership in the organization.