.
.
.
.
Chapter 3
The Human Brain
.
.
.
.
.
Mountcastle makes a similar observation. In a field of anatomists looking for minute differences in cortical regions, he shows that despite the differences, the neocortex is remarkably uniform. The same layers, cell types, and connections exist throughout. It looks like the six business cards everywhere. The differences are often so subtle that trained anatomists can’t agree on them. Therefore, Mountcastle argues, all regions of the cortex are performing the same operation. The thing that makes the vision area visual and the motor area motoric is how the regions of cortex are connected to each other and to other parts of the central nervous system.
In fact, Mountcastle argues that the reason one region of cortex looks slightly different from another is because of what it is connected to, and not because its basic function is different. He concludes that there is a common function, a common algorithm, that is performed by all the cortical regions. Vision is no different from hearing, which is no different from motor output. He allows that our genes specify how the regions of cortex are connected, which is very specific to function and species, but the cortical tissue itself is doing the same thing everywhere.
Let’s think about this for a moment. To me, sight, hearing, and touch seem very different. They have fundamentally different qualities. Sight involves color, texture, shape, depth, and form. Hearing has pitch, rhythm, and timbre. They feel very different. How can they be the same? Mountcastle says they aren’t the same, but the way the cortex processes signals from the ear is the same as the way it processes signals from the eyes. He goes on to say that motor control works on the same principle, too. Scientists and engineers have for the most part been ignorant of, or have chosen to ignore, Mountcastle’s proposal. When they try to understand vision or make a computer that can “see,” they devise vocabulary and techniques specific to vision. They talk about edges, textures, and three-dimensional representations. If they want to understand spoken language, they build algorithms based on rules of grammar, syntax, and semantics. But if Mountcastle is correct, these approaches are not how the brain solves these problems, and are therefore likely to fail. If Mountcastle is correct, the algorithm of the cortex must be expressed independently of any particular function or sense. The brain uses the same process to see as to hear. The cortex does something universal that can be applied to any type of sensory or motor system.
When I first read Mountcastle’s paper I nearly fell out of my chair. Here was the Rosetta stone of neuroscience-a single paper and a single idea that united all the diverse and wondrous capabilities of the human mind. It united them under a single algorithm. In one step it exposed the fallacy of all previous attempts to understand and engineer human behavior as diverse capabilities. I hope you can appreciate how radical and wonderfully elegant Mountcastle’s proposal is. The best ideas in science are always simple, elegant, and unexpected, and this is one of the best. In my opinion it was, is, and will likely remain the most important discovery in neuroscience. Incredibly, though, most scientists and engineers either refuse to believe it, choose to ignore it, or aren’t aware of it.
————–
Part of this neglect stems from a poverty of tools for studying how information flows within the six-layered cortex. The tools we do have operate on a grosser level and are generally aimed at locating where in the cortex, as opposed to when and how, various capabilities arise. For example, much of the neuroscience reported in the popular press these days implicitly favors the idea of the brain as a collection of highly specialized modules. Functional imaging techniques like functional MRI and PET scanning focus almost exclusively on brain maps and the functional regions I mentioned earlier. Typically in these experiments, a volunteer subject lies down with his or her head inside the scanner and performs some kind of mental or motor task. It might be playing a video game, generating verb conjugations, reading sentences, looking at faces, naming pictures, imagining something, memorizing lists, making financial decisions, and so on. The scanner detects which brain regions are more active than usual during these tasks and draws colored splotches over an image of the subject’s brain to pinpoint them. These regions are presumably central to the task. Thousands of functional imaging experiments have been done and thousands more will follow. Through the course of it all, we are gradually building up a picture of where certain functions happen in the typical adult brain. It is easy to say, “this is the face recognition area, this is the math area, this is the music area,” and so on. Since we don’t know how the brain accomplishes these tasks, it is natural to assume that the brain carries out the various activities in different ways.
But does it? A growing and fascinating body of evidence supports Mountcastle’s proposal. Some of the best examples demonstrate the extreme flexibility of the neocortex. Any human brain, if nourished properly and put in the right environment, can learn any of thousands of spoken languages. That same brain can also learn sign language, written language, musical language, mathematical language, computer languages, and body language. It can learn to live in frigid northern climes or in a scorching desert. It can become an expert in chess, fishing, farming, or theoretical physics. Consider the fact that you have a special visual area that seems to be specifically devoted to representing written letters and digits. Does this mean you were born with a language area ready to process letters and digits?
Unlikely. Written language is far too recent an invention for our genes to have evolved a specific mechanism for it. So the cortex is still dividing itself into task-specific functional areas long into childhood, based purely on experience. The human brain has an incredible capacity to learn and adapt to thousands of environments that didn’t exist until very recently. This argues for an extremely flexible system, not one with a thousand solutions for a thousand problems.
Neuroscientists have also found that the wiring of the neocortex is amazingly “plastic,” meaning it can change and rewire itself depending on the type of inputs flowing into it. For example, newborn ferret brains can be surgically rewired so that the animals’ eyes send their signals to the areas of cortex where hearing normally develops. The surprising result is that the ferrets develop functioning visual pathways in the auditory portions of their brains. In other words, they see with brain tissue that normally hears sounds. Similar experiments have been done with other senses and brain regions. For instance, pieces of rat visual cortex can be transplanted around the time of birth to regions where the sense of touch is usually represented. As the rat matures, the transplanted tissue processes touch rather than vision. Cells were not born to specialize in vision or touch or hearing.
Human neocortex is every bit as plastic. Adults who are born deaf process visual information in areas that normally become auditory regions. And congenitally blind adults use the rearmost portion of their cortex, which ordinarily becomes dedicated to vision, to read braille. Since braille involves touch, you might think it would primarily activate touch regions-but apparently no area of cortex is content to represent nothing. The visual cortex, not receiving information from the eyes like it is “supposed” to, casts around for other input patterns to sift through-in this case, from other cortical regions.
All this goes to show that brain regions develop specialized functions based largely on the kind of information that flows into them during development. The cortex is not rigidly designed to perform different functions using different algorithms any more than the earth’s surface was predestined to end up with its modern arrangement of nations. The organization of your cortex, like the political geography of the globe, could have turned out differently given a different set of early circumstances.
Genes dictate the overall architecture of the cortex, including the specifics of what regions are connected together, but within that structure the system is highly flexible.
Mountcastle was right. There is a single powerful algorithm implemented by every region of cortex. If you connect regions of cortex together in a suitable hierarchy and provide a stream of input, it will learn about its environment. Therefore, there is no reason for intelligent machines of the future to have the same senses or capabilities as we humans. The cortical algorithm can be deployed in novel ways, with novel senses, in a machined cortical sheet so that genuine, flexible intelligence emerges outside of biological brains.
————-
Let’s move on to a topic that is related to Mountcastle’s proposal and is equally surprising. The inputs to your cortex are all basically alike. Again, you probably think of your senses as being completely separate entities. After all, sound is carried as compression waves through air, vision is carried as light, and touch is carried as pressure on your skin. Sound seems temporal, vision seems mainly pictorial, and touch seems essentially spatial. What could be more different than the sound of a bleating goat versus the sight of an apple versus the feel of a baseball?
But let’s take a closer look. Visual information from the outside world is sent to your brain via a million fibers in your optic nerve. After a brief transit through the thalamus, they arrive at the primary visual cortex. Sounds are carried in via the thirty thousand fibers of your auditory nerve. They pass through some older parts of your brain and then arrive at your primary auditory cortex. Your spinal cord carries information about touch and internal sensations to your brain via another million fibers. They are received by your primary somatosensory cortex. These are the main inputs to your brain. They are how you sense the world.
You can visualize these inputs as a bundle of electrical wires or a bundle of optical fibers. You might have seen lamps made with optical fibers where pinpoints of colored light appear at the end of each fiber. The inputs to the brain are like this, but the fibers are called axons, and they carry neural signals called “action potentials” or “spikes,” which are partly chemical and partly electrical. The sense organs supplying these signals are different, but once they are turned into brain-bound action potentials, they are all the same-just patterns.
If you look at a dog, for example, a set of patterns will flow through the fibers of your optic nerve into the visual part of your cortex. If you listen to the dog bark, a different set of patterns will flow along your auditory nerve and into the hearing parts of your brain. If you pet the dog, a set of touch-sensation patterns will flow from your hand, through fibers in your spine, and into the parts of your brain that deal with touch. Each pattern-see the dog, hear the dog, feel the dog-is experienced differently because each gets channeled through a different path in the cortical hierarchy. It matters where the cables go to inside the brain. But at the abstract level of sensory inputs, these are all essentially the same, and are all handled in similar ways by the six-layered cortex. You hear sound, see light, and feel pressure, but inside your brain there isn’t any fundamental difference between these types of information. An action potential is an action potential. These momentary spikes are identical regardless of what originally caused them. All your brain knows is patterns.
.
.
.
.
Chapter 4
Memory
.
.
.
.
Computers have memory too, in the form of hard drives and memory chips; however, there are four attributes of neocortical memory that are fundamentally different from computer memory:
• The neocortex stores sequences of patterns.
• The neocortex recalls patterns auto-associatively.
• The neocortex stores patterns in an invariant form.
• The neocortex stores patterns in a hierarchy.
.
.
.
.
The next time you tell a story, step back and consider how you can only relate one aspect of the tale at a time. You cannot tell me everything that happened all at once, no matter how quickly you talk or I listen. You need to finish one part of the story before you can move on to the next. This isn’t only because spoken language is serial; written, oral, and visual storytelling all convey a narrative in a serial fashion. It is because the story is stored in your head in a sequential fashion and can only be recalled in the same sequence. You can’t remember the entire story at once. In fact, it’s almost impossible to think of anything complex that isn’t a series of events or thoughts.
You may have noticed, too, that in telling a story some people can’t get to the crux of it right away. They seem to ramble on with irrelevant details and tangents. This can be irritating. You want to scream, “Get to the point!” But they are chronicling the story as it happened to them, through time, and cannot tell it any other way.
Another example: I’d like you to imagine your home right now. Close your eyes and visualize it. In your imagination, go to the front door. Imagine what it looks like. Open your front door. Move inside. Now look to your left. What do you see? Look to the right. What is there? Go to your bathroom. What’s on the right? What’s on the left? What’s in the top right drawer? What items do you keep in your shower? You know all these things plus thousands more and can recall them in great detail. These memories are stored in your cortex. You might say these things are all part of the memory of your home. But you can’t think of them all at once. They are obviously related memories but there is no way you can bring to mind all of this detail at once. You have a thorough memory of your home; but to recall it you have to go through it in sequential segments, in much the same way as you experience it.
All memories are like this. You have to walk through the temporal sequence of how you do things. One pattern (approach the door) evokes the next pattern (go through the door), which evokes the next pattern (either go down the hall or ascend the stairs), and so on. Each is a sequence you’ve followed before. Of course, with a conscious effort I can change the order of how I describe my home to you. I can jump from basement to the second floor if I decide to focus on items in a nonsequential way. Yet once I start to describe any room or item I’ve chosen, I’m back to following .a sequence. Truly random thoughts don’t exist. Memory recall almost always follows a pathway of association.
You know the alphabet. Try saying it backward. You can’t because you don’t usually experience it backward. If you want to know what it’s like to be a child learning the alphabet, try saying it in reverse. That’s exactly what they’re confronted with. It’s really hard. Your memory of the alphabet is a sequence of patterns. It isn’t something stored or recalled in an instant or in an arbitrary order. The same thing goes for the days of the week, the months of the year, your phone number, and countless other things.
Your memory for songs is a great example of temporal sequences in memory. Think of a tune you know. I like to use “Somewhere over the Rainbow,” but any melody will suffice.
You cannot imagine the entire song at once, only in sequence. You can start at the beginning or maybe with the chorus, and then you play through it, filling in the notes one after another. You can’t recall the song backward,just as you can’t recall it all at once. You were first exposed to “Somewhere over the Rainbow” as it played through time, and you can only recall it in the same way you learned it.
This applies to very low level sensory memories too. Consider your tactile memory for textures. Your cortex has memories of what it feels like to hold a fistful of gravel, slide your fingers over velvet, and press down on a piano key. These memories are based on sequences every bit as much as the alphabet and songs are; it’s just that the sequences are shorter, spanning mere fractions of a second rather than many seconds or minutes. If! buried your hand in a bucket of gravel while you slept, when you woke up you wouldn’t know what you were touching until you moved your fingers. Your memory for the tactile texture of gravel is based on pattern sequences across the pressure and vibration-sensing neurons in your skin. These sequences are different from those you’d receive if your hand was buried in sand or Styrofoam pellets or dry leaves. As soon as you flexed your hand, the scraping and rolling of the pebbles would create the telltale pattern sequences of gravel and trigger the appropriate memory in your somatosensory cortex.
The next time you get out of the shower, pay attention to how you dry yourself off with a towel. I discovered that I dry myself off with nearly the exact same sequence of rubs, pats, and body positions each time. And via a pleasant experiment I discovered that my wife also follows a semirigid pattern when she steps out of the shower. You probably do too. If you follow a sequence, try changing it. You can will yourself to do it, but you need to stay focused. If your attention wanders, you’ll fall back into your accustomed pattern.
All memories are stored in the synaptic connections between neurons. Given the very large number of things we have stored in our cortex, and that at any moment in time we can recall only a tiny fraction of these stored memories, it stands to reason that only a limited number of synapses and neurons in your brain are playing an active role in memory recall at anyone time. As you start to recall what is in your home, one set of neurons becomes active, which then leads to another set of neurons being active, and so on. An adult human neocortex has an incredibly large memory capacity. But, even though we have stored so many things, we can only remember a few at any time and can only do so in a sequence of associations.
Here is a fun exercise. Try to recall details from your past, details of where you lived, places you visited, and people you knew. I find I can always uncover memories of things I haven’t thought of in many years. There are thousands of detailed memories stored in the synapses of our brains that are rarely used. At any point in time we recall only a tiny fraction of what we know. Most of the information is sitting there idly waiting for the appropriate cues to invoke it.
Computer memory does not normally store sequences of patterns. It can be made to do so using various software tricks (such as when you store a song on your computer), but computer memory does not do this automatically. In contrast, the cortex does store sequences automatically. Doing so is an inherent aspect of the neocortical memory system.
————–
Now let’s consider the second key feature of our memory, its auto-associative nature. As we saw in chapter 2, the term simply means that patterns are associated with themselves. An auto-associative memory system is one that can recall complete patterns when given only partial or distorted inputs. This can work for both spatial and temporal patterns. If you see your child’s shoes sticking out from behind the draperies, you automatically envision his or her entire form. You complete the spatial pattern from a partial version of it. Or imagine you see a person waiting for a bus but can only see part of her because she is standing partially behind a bush. Your brain is not confused. Your eyes only see parts of a body, but your brain fills in the rest, creating a perception of a whole person that’s so strong you may not even realize you’re only inferring.
You also complete temporal patterns. If you recall a small detail about something that happened long ago, the entire memory sequence can come flooding back into your mind. Marcel Proust’s famous series of novels, Remembrance of Things Past, opened with the memory of how a madeleine cookie smelled and he was off and running for a thousand-plus pages. During conversation we often can’t hear all the words if we are in a noisy environment. No problem. Our brains fill in what they miss with what they expect to hear. It’s well established that we don’t actually hear all the words we perceive. Some people complete others’ sentences aloud, but in our minds all of us are doing this constantly. And not just the ends of sentences, but the middles and beginnings as well. For the most part we are not aware that we’re constantly completing patterns, but it’s a ubiquitous and fundamental feature of how memories are stored in the cortex. At any time, a piece can activate the whole. This is the essence of auto-associative memories.
Your neocortex is a complex biological auto-associative memory. During each waking moment, each functional region is essentially waiting vigilantly for familiar patterns or pattern fragments to come in. You can be in deep thought about something, but the instant your friend appears your thoughts switch to her. This switch isn’t something you chose to do. The mere appearance of your friend forces your brain to start recalling patterns associated with her. It’s unavoidable. After an interruption we frequently have to ask, “What was I thinking about?” A dinner conversation with friends follows a circuitous route of associations. The talk may start with the food in front of you, but the salad evokes an associated memory of your mother’s salad at your wedding, which leads to a memory of someone else’s wedding, which leads to a memory of where they went on their honeymoon, to the political problems in that part of the world, and so on. Thoughts and memories are associatively linked, and again, random thoughts never really occur. Inputs to the brain auto-associatively link to themselves, filling in the present, and auto-associatively link to what follows next. We call this chain of memories thought, and although its path is not deterministic, we are not fully in control of it either.
.
.
.
.
Chapter 5
A New Framework of Intelligence
.
.
.
.
Mammals evolved a large neocortex because it gave them some survival advantage, and such an advantage must ultimately be rooted in behavior. But in the beginning, the cortex served to make more efficient use of existing behaviors, not to create entirely new behaviors. To make the case clear, we need to take a look at how our brains evolved.
Simple nervous systems emerged not long after multicellular creatures started squiggling all over the Earth, hundreds of millions of years ago, but the story of real intelligence begins more recently with our reptilian forebears. The reptiles were successful in their conquest of the land. They spread over every continent and diversified into numerous species. They had keen senses and well-developed brains that endowed them with complex behavior. Their direct descendants, today’s surviving reptiles, still have them. An alligator, for example, has sophisticated senses just like you and me. It has well-developed eyes, ears, nose, mouth, and skin. It carries out complex behaviors including the ability to swim, run, hide, hunt, ambush, sun, nest, and mate.
What is the difference between a human brain and a reptile brain? A lot and a little. I say a little because, to a rough approximation, everything in a reptile’s brain exists in a human brain. I say a lot because a human brain has something really important that a reptile does not have: a large cortex. You sometimes hear people refer to the “old” brain or the “primitive” brain. Every human has these more ancient structures in the brain,just like a reptile. They regulate blood pressure, hunger, sex, emotions, and many aspects of movement. When you stand, balance, and walk, for example, you are relying heavily on the old brain. If you hear a lightening sound, panic, and start to run, that is mostly your old brain. You don’t need more than a reptile brain to do a lot of interesting and useful things. So what does the neocortex do if it isn’t strictly required to see, hear, and move?
Mammals are more intelligent than reptiles because of their neocortex. (The word itself is derived from the Latin words for “new bark” or “new rind,” because the cortex literally covers the old brain.) The neocortex first appeared tens of millions of years ago and only mammals have one. What makes humans smarter than other mammals is primarily the large area of our neocortex-which expanded dramatically only a couple of million years ago. Remember, the cortex is built using a common repeated element. The human cortical sheet is the same thickness and has very nearly the same structure as the cortex in our mammal relatives. When evolution makes something big very quickly, as it did with human cortex, it does so by copying an existing structure. We got smart by adding many more elements of a common cortical algorithm. There is a common misconception that the human brain is the pinnacle of billions of years of evolution. This may be true if we think of the entire nervous system. However, the human neocortex itself is a relatively new structure and hasn’t been around long enough to undergo much long-term evolutionary refinement.
Here then is the core of my argument on how to understand the neocortex, and why memory and prediction are the keys to unlocking the mystery of intelligence. We start with the reptilian brain with no cortex. Evolution discovers that if it tacks on a memory system (the neocortex) to the sensory path of the primitive brain, the animal gains an ability to predict the future. Imagine the old reptilian brain is still doing its thing, but now sensory patterns are simultaneously fed into the neocortex. The neocortex stores this sensory information in its memory. At a future time when the animal encounters the same or a similar situation, the memory recognizes the input as similar and recalls what happened in the past. The recalled memory is compared with the sensory input stream. It both “fills in” the current input and predicts what will be seen next. By comparing the actual sensory input with recalled memory, the animal not only understands where it is but can see into the future.
Now imagine that the cortex not only remembers what the animal has seen but also remembers the behaviors the old brain performed when it was in a similar situation. We don’t even have to assume the cortex knows the difference between sensations and behavior; to the cortex they are both just patterns. When our animal finds itself in the same or a similar situation, it not only sees into the future but recalls which behaviors led to that future vision. Thus, memory and prediction allow an animal to use its existing (old brain) behaviors more intelligently.
For example, imagine you’re a rat learning to navigate a maze for the first time. Aroused by uncertainty or hunger, you will use the skills inherent to your old brain to explore the new environment-listening, looking, sniffing, and creeping close to the walls. All this sensory information is used by your old brain but is also passed up to your neocortex, where it is stored. At some future time, you find yourself in the same maze. Your neocortex will recognize the current input as one it has seen before and recall the stored patterns representing what happened in the past. In essence, it allows you to see a short way into the future. If you were a talking rat, you might say, “Oh, I recognize this maze, and I remember this corner.” As your neocortex recalls what happened in the past, you will envision finding the cheese you saw last time you were in the maze, and how you got to it. “In turn right here, I know what.will happen next. There’s a piece of cheese down at the end of this hallway. I see it in my imagination.” When you scurry through the maze, you rely on older, primitive structures to carry out movements like lifting your feet and sweeping your whiskers. With your (relatively) big neocortex, you can remember the places you have been, recognize them again in the future, and make predictions about what will happen next. A lizard without a neocortex has a much poorer ability to remember the past and may have to search a maze anew every time. You (the rat) understand the world and the immediate future because of your cortical memory. You see vivid images of the rewards and dangers that lie ahead of each decision, and so you move more effectively through your world. You can literally see the future.
But notice you are not performing any particularly complex or fundamentally new behaviors. You are not building yourself a hang glider and flying to the cheese at the end of the hallway. Your neocortex is forming predictions about sensory patterns that allow you to see into the future, but your palette of available behaviors is pretty much unaffected. Your ability to scurry, clamber, and explore is still a lot like that of a lizard.
As the cortex got larger over evolutionary time, it was able to remember more and more about the world. It could form more memories, and make more predictions. The complexity of those memories and predictions also increased. But something else remarkable happened that led to the uniquely human abilities for intelligent behavior.
Human behavior transcends the old basic repertoire of moving around with ratlike skills. We have taken neocortical evolution to a new level. Only humans create written and spoken language. Only humans cook their food, sew clothes, fly planes, and build skyscrapers. Our motor and planning abilities vastly exceed those of our closest animal relatives. How can the cortex, which was designed to make sensory predictions, generate the incredibly sophisticated behavior unique to humans? And how could this superior behavior evolve so suddenly? There are two answers to this question. One is that the neocortical algorithm is so powerful and flexible that with a little bit of rewiring, unique to humans, it can create new, sophisticated behaviors. The other answer is that behavior and prediction are two sides of the same thing. Although the cortex can envision the future, it can make accurate sensory predictions only if it knows what behaviors are being performed.
In the simple example of the rat looking for the cheese, the rat remembers the maze and uses this memory to predict that it will see the cheese around the corner. But the rat could turn left or turn right; only by simultaneously remembering the cheese and the correct behavior, “turn right at the fork,” can the rat make the prediction of the cheese come true. Although this is a trivial example, it gets to the essence of how sensory prediction and behavior are intimately related. All behavior changes what we see, hear, and feel. Most of what we sense at any moment is highly dependent on our own actions. Move your arm in front of your face. To predict seeing your arm, your cortex has to know that it has commanded the arm to move. If the cortex saw your arm moving without the corresponding motor command, you would be surprised. The simplest way to interpret this would be to assume your brain first moves the arm and then predicts what it will see. I believe this is wrong. Instead I believe the cortex predicts seeing the arm, and this prediction is what causes the motor commands to make the prediction come true. You think first, which causes you to act to make your thoughts come true.
.
.
.
.
Chapter 6
How the Cortex Works
.
.
.
.
Let me describe this using another mental picture. Imagine two pieces of paper with lots of little holes punched in them. The holes on one paper represent the columns that have active layer 2 or layer 3 cells, our invariant prediction. The holes on the other paper represent columns with partial input from below. If you put one piece of paper on top of the other, some of the holes will line up, others won’t. The holes that line up represent the columns we think should be active.
This mechanism not only makes specific predictions, it also resolves ambiguities from the sensory inputs. Very often the input to a region of cortex will be ambiguous, as we saw with the colored papers, or when you hear a semi-garbled word. This bottom-up/ top-down matching mechanism enables you to decide between two or more interpretations. And once you decide, you relay your interpretation to the region below.
Every moment in your waking life, each region of your neocortex is comparing a set of expected columns driven from above with the set of observed columns driven from below. Where the two sets intersect is what we perceive. If we had perfect input from below and perfect predictions, then the set of perceived columns would always be contained in the set of predicted columns. We often don’t have such agreement. The method of combining partial prediction with partial input resolves ambiguous input, it fills in missing pieces of information, and it decides between alternative views. It is how we combine an expected pitch-invariant interval with the last heard note to predict the next specific note in a melody. It is how we decide whether a picture is of a vase or of two faces. It is how we split our motor stream either to write or to speak the Gettysburg Address.
Finally, in addition to projecting to lower cortical regions, layer 6 cells can send their output back into layer 4 cells of their own column. When they do, our predictions become the input. This is what we do when daydreaming or thinking. It allows us to see the consequences of our own predictions. We do this many hours a day as we plan the future, rehearse speeches, and worry about events to come. Longtime cortical modeler Stephen Grossberg calls this “folded feedback.” I prefer “imagining.”
————-
One last topic before we leave this section. I have pointed out several times that most often what we see, hear, or feel is highly dependent on our own actions. What we see is dependent on where our eyes saccade and how we turn our heads. What we feel is dependent on how we move our limbs and fingers. What we hear is sometimes dependent on what we say and do.
Therefore, to predict what we will sense next, we have to know what actions we are undertaking. Motor behavior and sensory perception are highly interdependent. How can we make predictions if what we sense next is largely a result of our own actions? Fortunately, there is a surprising and elegant solution to this problem, although many of the details are not understood.
The first surprising discovery is that perception and behavior are almost one and the same. As I mentioned earlier, most if not all regions of the cortex, even visual areas, participate in the creation of movement. The layer 5 cells that project to the thalamus and then to layer 1 also seem to have a motor function because they simultaneously project to motor areas of the old brain. Thus, the knowledge of ”what just happened”-both sensory and motor-is available in layer 1.
The second surprising thing, and a consequence of the first, is that motor behavior must also be represented in a hierarchy of invariant representations. You generate the movements necessary to carry out a particular action by thinking of doing it in a detail-invariant form. As the motor command travels down the hierarchy, it gets translated into the complex and detailed sequences required to perform the activity you expected to do. This is happening in both “motor” cortex and “sensory” cortex, which blurs the distinction between the two. If region IT of visual cortex is perceiving “nose,” the mere act of switching to the representation for “eye” will generate the saccade necessary: to make this prediction a reality. The particular saccade necessary to move from seeing a nose to seeing an eye varies depending on where the face is. A close face requires a larger saccade; a more distant face requires a smaller saccade. A tilted face requires saccading at an angle different from the one for a level face. The details of the needed saccade are determined as the prediction of seeing the “eye” moves toward V1. The saccade becomes increasingly specific the farther down it goes, resulting in a saccade that lands your foveas right on target, or pretty close.
Let’s look at another example. For me to physically move from my living room to my kitchen, all my brain has to do is mentally switch from the invariant representation of my living room to the invariant representation of my kitchen. This switch causes a complex unfolding of sequences. The process of generating the sequence of predictions of what I will see, feel, and hear while walking from the living room to the kitchen also generates the sequence of motor commands that makes me walk from my living room to my kitchen and move my eyes as I do so. Prediction and motor behavior work hand in hand as patterns flow down and up the cortical hierarchy. As strange as it sounds, when your own behavior is involved, your predictions not only precede sensation, they determine sensation. Thinking of going to the next pattern in a sequence causes a cascading prediction of what you should experience next. As the cascading prediction unfolds, it generates the motor commands necessary to fulfill the prediction. Thinking, predicting, and doing are all part of the same unfolding of sequences moving down the cortical hierarchy.
“Doing” by thinking, the parallel unfolding of perception and motor behavior, is the essence of what is called goal oriented behavior. Goal-oriented behavior is the holy grail of robotics. It is built into the fabric of the cortex.
We can turn off our motor behavior, of course. I can think of seeing something without actually seeing it and I can think of going to my kitchen without actually doing so. But thinking of doing something is literally the start of how we do it.
.
.
.
.
How the Cortex Learns
.
.
.
.
When you are born your cortex essentially doesn’t know anything. It doesn’t know about your language, your culture, your home, your town, songs, the people you will grow up with, nothing. All this information, the structure of the world, has to be learned. The two basic components of learning are forming the classifications of patterns and building sequences. These two complementary memory components interact. As one region learns sequences, the inputs it sends to the layer 4 cells in higher cortical regions change. These layer 4 cells therefore learn to form new classifications, which changes the pattern projected back to layer 1 in the lower region, which affects the sequences.
The basics of forming sequences is to group patterns together that are part of the same object. One way to do this is by grouping patterns that occur contiguously in time. If a child holds a toy in her hand and slowly moves it, her brain can safely assume that the image on her retina is of the same object moment to moment, and therefore the changing set of patterns can be grouped together. At other times you need outside instruction to help you decide which patterns belong together. To learn that apples and bananas are fruits, but carrots and celery are not, requires a teacher to guide you to group these items as fruits. Either way, your brain slowly builds sequences of patterns that belong together. But as a region of cortex builds sequences, the input to the next region changes. The input changes from representing mostly individual patterns to representing groups of patterns. The input to a region changes from notes to melodies, from letters to words, from noses to faces, and so on. Because the bottom-up inputs to a region become more “object-oriented,” the higher region of cortex can now learn sequences of these higher-order objects. Where before a region built sequences of letters, it now builds sequences of words. The unexpected result of this learning process is that, during repetitive learning, representations of objects move down the cortical hierarchy. During the early years of your life, your memories of the world first form in higher regions of cortex, but as you learn they are re-formed in lower and lower parts of the cortical hierarchy. It isn’t that the brain moves them; it has to relearn them over and over. (I am not suggesting that all memories start at the top of the cortex. The actual formation of memories is more complex. I believe layer 4 pattern classification starts at the bottom and moves up. But as it does, we start forming sequences that then move down. It is the memory of sequences I am suggesting re-form lower and lower in the cortex.) As simple representations move down, the regions at the top are able to learn more complex and subtle patterns.
You can observe the creation and downward movement of hierarchical memory by observing how a child learns. Consider how we learn to read. The first thing we learn is to recognize individual printed letters. This is a slow and difficult task requiring conscious effort. Then we move on to recognizing simple words. Again, it is difficult and slow at first, even for three-letter words. The child can read each letter in sequence and sound out the letters one after another, but it takes a fair amount of practice before the word itself is recognized as a word. After learning simple words, we struggle with complex, multisyllable words. At first, we sound out each syllable, concatenating them as we did with letters when learning simple words. After years of practice, a person can read quickly. We get to the point where we don’t actually see all the individual letters but instead recognize entire words and often entire phrases at a glance. It isn’t just that we are faster; we are actually recognizing words and phrases as entities. When we read an entire word at one time, do we still see the letters? Yes and no. Obviously, the retina sees the letters and therefore so do regions of V1. But the recognition of the letters is occurring fairly low in the cortical hierarchy, say in V2 or V4. By the time the signal gets to IT, the individual letters are no longer represented. What at first took the effort of your entire visual cortex-recognizing individual letters-is now occurring closer to the sensory input. As memory of simple objects like letters moves down the hierarchy, the higher regions have the ability to learn complex objects like words and phrases.
Learning to read music is another example. At first you have to concentrate on every note. With practice, you start to recognize common note sequences, then entire phrases. After much practice, it is as if you don’t see most of the notes at all. The sheet music is there only to remind you of the major structure of the piece; the detailed sequences have been memorized lower down. This type of learning occurs in both motor and sensory areas.
A young brain is slower to recognize inputs and slower to make motor commands because the memories used in these tasks are higher up the cortical hierarchy. Information has to flow all the way up and down, maybe with multiple passes, to resolve conflicts. It takes time for the neural signals to travel up and down the cortical hierarchy. A young brain also has not yet formed complex sequences at the top and therefore cannot recognize and play back complex patterns. A young brain cannot understand the higher-order structure of the world. Compared to an adult’s, a child’s language is simple, his music is simple, and his social interactions are simple.
If you study a particular set of objects over and over, your cortex re-forms memory representations for those objects down the hierarchy. This frees up the top for learning more subtle, more complex relationships. According to the theory, this is what makes an expert.
In my work designing computers, some people are surprised by how quickly I can look at a product and see the problems inherent in its design. After twenty-five years of designing computers, I have a better-than-average model of the issues associated with mobile computing devices. Similarly, an experienced parent can easily recognize why his child is upset, whereas a first-time parent may struggle with how to handle a situation. An experienced business manager can readily see the flaws and advantages of the structure of an organization whereas the novice manager just can’t understand these things. They have the same input, but the novice’s model is not as sophisticated. In all such cases and a thousand more, we start by learning the basics, the simplest structure. Over time we move our knowledge down the cortical hierarchy and, therefore, we have the opportunity at the top for learning higher-order structure. It is this higher-order structure that makes us experienced. Experts and geniuses have brains that see structure of structure and patterns of patterns beyond what others do. You can become expert by practice, but there certainly is a genetic component to talent and genius too.
.
.
.
.
Chapter 7
Consciousness and Creativity
.
.
.
.
Imagine you are about to have dinner in an unfamiliar restaurant and you want to wash your hands. Even though you have never been in this building before, your brain predicts that there will be a restroom somewhere in the restaurant with a basin suitable for hand washing. How does it know this? Other restaurants you have been in have a restroom, and by analogy this restaurant will likely have one, too. Further, you know where and what to look for. You predict there will be a door or sign with some type of symbol associated with men or women. You predict it will be toward the back of the restaurant, either by the bar or down a hall, but generally not in plain view of the eating areas. Again, you have never been in this particular restaurant before, but by analogy to other eating establishments you are able to find what you need. You don’t look around randomly. You look for expected patterns that let you find the restroom quickly. This kind of behavior is a creative act; it is predicting the future by analogy to the past. We don’t normally think of this as being creative, but it very much is.
Recently I bought a vibraphone. We have a piano, but I had never played the vibraphone before. The day we brought it home, I took a sheet of music from the piano, placed it on the stand over the vibraphone, and started playing simple melodies. My ability to do this was not remarkable. But in a fundamental way, it was a creative act. Think about what was involved. I have an instrument that is very different from a piano. The vibraphone has gold metal bars; the piano has black and white keys. The gold bars are big and gradually change in size; the keys are small and of two different sizes. The gold bars are arranged in two different rows; the black and white keys are interleaved. On one instrument I use my fingers, and on the other I swing mallets. For this I’m standing up, and for that I’m sitting down. The particular muscles and motions needed to play the vibraphone are completely different from those needed to play the piano.
So how was I able to play a melody on an unfamiliar instrument? The answer is that my cortex sees an analogy between the keys on a piano and the bars on a vibraphone. Using this similarity allowed me to play a tune. It isn’t really any different from singing a song in a new key. In both cases, we know what to do by analogy to past learning. I realize that to you the similarity between these two instruments may appear obvious, but that is only because our brains automatically see analogies. Try to program a computer to find similarities between objects such as pianos and vibraphones and you will see how incredibly difficult this is. Prediction by analogy-creativity-is so pervasive we normally don’t notice it.
We do, however, believe we are being creative when our memory-prediction system operates at a higher level of abstraction, when it makes uncommon predictions, using uncommon analogies. For example, most people would agree that a mathematician who proves a difficult conjecture is being creative. But let’s take a close look at what’s involved with her mental efforts. Our mathematician stares hard at an equation and says, “How am I going to tackle this problem?” If the answer isn’t readily obvious she may rearrange the equation. By writing it down in a different fashion, she can look at the same problem from a different perspective. She stares some more. Suddenly she sees a part of the equation that looks familiar. She thinks, “Oh, I recognize this. There’s a structure to this equation that is similar to the structure of another equation I worked on several years ago.” She then makes a prediction by analogy. “Maybe I can solve this new equation using the same techniques I used successfully on the old equation.” She is able to solve the problem by analogy to a previously learned problem. It is a creative act.
My father had a mysterious blood disorder that his physicians could not diagnose. So how did they know what treatment to offer? One of the things they did was to look at months of data taken from analyses of my father’s blood to see if they could identify patterns. (My father printed a beautiful chart so the doctors could see the data clearly.) While his symptoms did not closely match those of known diseases, there were some similarities. The doctors ended up basing his treatment on a mixture of strategies that had worked for other blood disorders. The treatments used were guesses based on analogies to diseases the physicians had previously treated. Recognizing these patterns required extensive exposure to other uncommon diseases.
Shakespeare’s metaphors are the paragon of creativity. “Love is a smoke made with the fume of sighs.” ”Adversity’s sweet milk, philosophy.” “There’s daggers in men’s smiles.” Such metaphors become obvious when you see them but they’re very hard to invent, which is one reason why Shakespeare is regarded as a literary genius. To create such metaphors he had to see a succession of clever analogies. When he writes “There’s daggers in men’s smiles,” he is not talking about daggers or smiles. Daggers are analogous to ill intent, and men’s smiles are analogous to deceit. Two clever analogies in only five words! At least that is how I interpret it. Poets have the gift of correlating seemingly unrelated words or concepts in manners that illuminate the world in new ways. They create unexpected analogies as a means of teaching higher-level structure.
In fact, highly creative works of art are appreciated because they violate our predictions. When you see a film that breaks the familiar mold of a character, story line, or cinematography (including special effects), you like it because it is not the same old same old. Paintings, music, poetry, novels-all creative artistic forms-strive to break convention and violate the expectations of an audience. There is a contradictory tension in what makes a work of art great. We want art to be familiar yet at the same time to be unique and unexpected. Too much familiarity is retread or kitsch; too much uniqueness is jarring and difficult to appreciate. The best works break some expected patterns while simultaneously teaching us new ones. Consider a great piece of classical music. The best music has an appeal at a simple level-good beat, simple melody and phrasing. Anyone can understand and appreciate it. However, it is also a little different and unexpected. But the more you listen to it, the more you see there is pattern in the unexpected parts, such as repeated unusual harmonies or key changes. The same is true with great literature or great movies. The more you read or see them, the more creative detail and complexity of structure you observe.
You’ve probably had the experience of looking at something when a twinge of recognition goes off in your head: “Hmmm, I’ve seen this pattern before, someplace else. . .” You may not have been trying to solve a problem, it’s just that an invariant representation in your brain was activated by a novel situation. You saw an analogy between two normally unrelated events. I might recognize that promoting a scientific idea is similar to selling a business idea or that bringing about political reform is like raising children. If I’m a poet, voila!, I have a new metaphor. If I’m a scientist or engineer, I have a new solution to a long standing problem. Creativity is mixing and matching patterns of everything you’ve ever experienced or come to know in your lifetime. It’s saying “this is kinda like that.” The neural mechanism for doing this is everywhere in the cortex.
Are Some People More Creative than Others?
A related question I often hear is, “If all brains are inherently creative, why are there differences in our creativity?” The memory prediction framework points to two possible answers. One has to do with nature and the other with nurture.
On the nurture side, everyone has different life experiences.Therefore everyone develops different models and memories of the world in his or her cortex, and will make different analogies and predictions. If I have been exposed to music, I will be able to sing songs in new keys and play simple melodies on new instruments. If! have never been exposed to music, I will not be able to make these predictive leaps. If I have studied physics, I will be able to explain the behavior of everyday objects via analogy to the laws of physics. If! grew up with dogs, I am apt to see analogies about dogs and will be better at predicting their behavior. Some people are more creative in social situations or in language, math, or diplomacy, all based on the environment they grew up in. Our predictions, and thus our talents, are built upon our experiences.
In chapter 6, I described how memories are pushed down the cortical hierarchy. The more you are exposed to certain patterns, the more the memory of these patterns are re-formed at lower levels. This allows you to learn the relationships among higher-order abstract objects at the top. It’s the essence of expertise. An expert is someone who through practice and repeated exposure can recognize patterns that are more subtle than can be recognized by a nonexpert, such as the shape of a fin on a late-fifties car or the size of a spot on a seagull’s beak. Experts can recognize patterns on top of patterns. Ultimately there is a physical limit to what we can learn constrained by the size of our cortex. But as humans, our cortex is large compared to other species and we have a tremendous flexibility in what we can learn. It all depends on what we are exposed to throughout our lives.
.
.
.
.
We continued this line of argument-a yes you are, no I’m not kind of thing-until it was time to head up to dinner. I don’t think I changed anyone’s mind about the existence and meaning of consciousness. But I was trying to get them to realize that most people think consciousness is some kind of magical sauce that is added on top of the physical brain. You’ve got a brain, made of cells, and you pour consciousness, this magical sauce, on it, and that’s the human condition. In this view, consciousness is a mysterious entity separate from brains. That’s why zombies have brains but they don’t have consciousness. They have all the mechanical stuff, neurons and synapses, but they don’t have the special sauce. They can do everything a human can do. From the outside you can’t tell a zombie from a human.
The idea that consciousness is something extra stems from earlier beliefs in elan vital-a special force once thought to animate living things. People believed you needed this life force to explain the difference between rocks and plants or metals and maidens. Few people believe this anymore. Nowadays we know enough about the differences between inanimate and animate matter to understand that there isn’t a special sauce. We now know a great deal about DNA, protein folding, gene transcription, and metabolism. While we don’t yet know all the mechanisms of living systems, we know enough about biology to leave out magic. Similarly, no longer do people suggest it takes magic or spirits to make muscles move. We have folding proteins that pull long molecules past one another. You can read all about it.
Nevertheless, many people persist in believing that consciousness is different and can’t be explained in reductionist biological terms. Again, I am not a student of consciousness. I haven’t read all the philosophers’ opinions. But I have some ideas about what I think. people are confusing in this debate. I believe consciousness is simply what it feels like to have a neocortex. But we can do better than that. We can break consciousness into two major categories. One is similar to self-awareness-the everyday notion of being conscious. This is relatively easy to understand. The second is qualia-the idea that feelings associated with sensation are somehow independent of sensory input. Qualia is the harder part.
When most people say the word conscious, they are referring to the first category. “Were you conscious that you walked past me without saying hello?” “Were you conscious when you fell out of bed last night?” “You aren’t conscious when you sleep.” Some people say this form of consciousness is exactly the same as awareness. The two are close, but I don’t think. awareness quite captures it correctly. I suggest this meaning of consciousness is synonymous with forming declarative memories. Declarative memories are memories that you can recall and talk about to someone else. You can express them verbally. If you ask me where I went last weekend, I can tell you. That is a declarative memory. If you ask me how to balance a bicycle, I can tell you to hold the handle bar and push the pedals, but I can’t explain exactly how to do it. How to balance a bicycle has mostly to do with neural activity in the old brain, so it is not a declarative memory.
I have a little thought experiment to show how our everyday notion of consciousness is the same as forming declarative memories. Recall that all memory is believed to reside in physical changes to synapses and the neurons they connect to. Therefore, if I had a method to reverse those physical changes, your memory would be erased. Now imagine I could flip a switch and return your brain to the exact physical state it was in at some point in the past. It could be an hour ago, twenty-four hours ago, whatever. I just flip the switch in my way-back machine and your synapses and neurons return to a previous state in time. By doing so, I erase all your memory of what occurred since that time.
Let’s assume you go through today and wake up tomorrow. But just as you’re waking up, I flip the switch and erase the last twenty-four hours. You would have absolutely zero memory of the previous day. From your brain’s perspective, yesterday never happened. I would tell you it’s Wednesday and you’d protest, “No, it’s Tuesday. I’m certain of it. The calendar has been altered. No way, this is Tuesday. Why are you pulling this trick on me?” But everyone whom you had met on Tuesday would say that you had been conscious throughout the day. They saw you, had lunch with you, and talked with you. Don’t you remember it? You’d say no, it didn’t happen. Finally, shown a video of you having lunch, you gradually become convinced that the day did happen, even though you have no memory of it. It’s as if you were a zombie for a day, not conscious. However, you were conscious at the time. Your belief that you were conscious disappeared only when your declarative memory was erased.
This thought experiment captures the equivalence between declarative memory and our everyday notion of being conscious. If during and at the end of a game of tennis I ask you if you are conscious, you would, of course, say yes. If! then erased your memory of the last two hours, you would claim to have been unconscious and not responsible for your actions during that time. In either case, you played the same game of tennis. The only difference is whether you have a memory of it at the time I ask you. Therefore, this meaning of consciousness is not absolute. It can be changed after the fact by memory erasure.
The more difficult question about consciousness concerns qualia. Qualia is often phrased in Zen-like queries, such as “Why is red red and green green? Does red look the same to me as it does to you? Why is red emotionally laden with certain feelings? It has a certain inextricable quality or feelingness to me. What feelingness does it cause in you?”
I find such descriptions difficult to relate to neurobiology, so I’d like to rephrase the question. For me, an equivalent question, but one I still find hard to explain, is, Why do different senses seem qualitatively different? Why does sight seem different from hearing and why does hearing seem different from touch? If the cortex is the same everywhere, if it works with the same processes, if it is just dealing with patterns, if no sound or light enters the brain,just patterns, then why does vision seem so different from hearing? I find it difficult to describe how sight differs from hearing, but it self-evidently is. I assume it is for you, too. Yet an axon representing sound and another representing light are, for all practical purposes, identical. “Lightness” and “soundness” are not carried down the axon of a sensory neuron.
People with a condition called synesthesia have brains that blur the distinction between the senses-certain sounds have a color, or certain textures have a color. This tells us that the qualitative aspect of a sense is not immutable. Through some sort of physical modification, a brain can impart a qualitative aspect of vision to an auditory input.
So what is the explanation for qualia? I can think of two possibilities, neither of which I find completely satisfactory. One is that although hearing, touch, and vision work under similar principles in the neocortex, they are handled differently below the cortex. Hearing relies on a set of audition-specific subcortical structures that process auditory patterns before they reach the cortex. Somatosensory patterns also travel through a set of subcortical areas that are unique to somatic senses. Perhaps qualia, like emotions, are not mediated purely by the neocortex. If they are somehow bound up with subcortical parts of the brain that have unique wiring, perhaps tied to emotion centers, this might explain why we perceive them differently, even if it doesn’t help explain why there is any sort of qualia sensation in the first place.
The other possibility I can think of is that the structure of the inputs-differences in the patterns themselves-dictates how you experience qualitative aspects of the information. The nature of the spatial-temporal pattern on the auditory nerve is different from the nature of the spatial-temporal pattern on the optic nerve. The optic nerve has a million fibers and carries quite a bit of spatial information. The auditory nerve has only thirty thousand fibers and carries more temporal information. These differences may be related to what we call qualia.
We can be certain that however consciousness is defined, memory and prediction play crucial roles in creating it.
Related to consciousness are the notions of mind and soul. As a child I used to wonder what it would have been like if “I” had been born in another child’s body in another country, as if “I” was somehow independent of my body. These feelings of a mind independent of physicalness are common and a natural consequence of how the neocortex works. Your cortex creates a model of the world in its hierarchical memory. Thoughts are what occur when this model runs on its own; memory recall leads to predictions, which act like sensory inputs, which lead to new memory recall, and so on. Our most contemplative thoughts are not driven by or even connected to the real world; they are purely a creation of our model. We close our eyes and seek quiet so that our thinking will not be interrupted by sensory input. Of course our model was originally created by exposure to the real world through our senses, but when we plan and think about the world, we do so via the cortical model, not the world itself.
To the cortex, our bodies are just part of the external world. Remember, the brain is in a quiet and dark box. It knows about the world only via the patterns on the sensory nerve fibers. From the brain’s perspective as a pattern device, it doesn’t know about your body any differently than it knows about the rest of the world. There isn’t a special distinction between where the body ends and the rest of the world begins. But the cortex has no ability to model the brain itself because there are no senses in the brain. Thus we can see why our thoughts appear independent of our bodies, why it feels like we have an independent mind or soul. The cortex builds a model of your body but it can’t build a model of the brain itself. Your thoughts, which are located in the brain, are physically separate from the body and the rest of the world. Mind is independent of body, but not of brain.
We can clearly see this differentiation through trauma and disease. If someone loses a limb, his brain’s model of the limb may nevertheless remain intact, resulting in a so-called phantom limb, which he can still feel attached to his body. On the flip side, if he suffers cortical trauma he may lose his model of the arm even though he retains the arm itself. In this case he may suffer what’s known as alien limb syndrome and have the uncomfortable, perhaps intolerable, feeling that the arm is not his own and is being controlled by someone else. Some even insist that the limb should be amputated! If our brain stays intact while the rest of our body becomes ill, we have the feeling of a healthy mind trapped in a dying body, although what we really have is a healthy brain trapped in a dying body. It is natural to imagine that our mind will continue after the death of our body, but when the brain dies so does the mind. The truth of this is evident if our brains fail before our bodies. People with Alzheimer’s disease or with serious brain damage lose their minds even if their bodies stay healthy.
What is Imagination?
Conceptually, imagination is rather simple. Patterns flow into each cortical area either from your senses or from lower areas of the memory hierarchy. Each cortical area creates predictions, which are sent back down the hierarchy. To imagine something, you merely let your predictions turn around and become inputs. Without physically doing anything, you can follow the consequences of your predictions. “If this happens, then this will happen, then this will happen,” and so on. We do this when preparing for a business meeting, playing a game of chess, preparing for a sports event, or doing a thousand other things.
In chess you imagine moving your knight to a certain position and then visualize what the board will look like after the move. With this image in mind, you predict what your opponent will do and what the board will look like following that move. Then you predict what you will do, and so on. You walk through the imagined steps and their consequences. Ultimately you decide, based on this imagined sequence of events, whether the initial move was a good one or not. Certain athletes, such as downhill skiers, can improve their performance if they mentally rehearse the racecourse over and over in their head. By closing their eyes and imagining each and every turn, every obstacle, and even being on the winning stand, they increase their chances of success. Imagining is just another word for planning. This is where the predictive ability of our cortex pays off. It permits us to know what the consequences of our actions will be before we do them.
Imagining requires a neural mechanism for turning a prediction into an input. In chapter 6 I proposed that cells in layer 6 are where precise prediction occurs. Cells in this layer project down to lower levels of the hierarchy, but they also project back up to the input cells in layer 4. Thus a region’s outputs can become its own inputs. As I mentioned earlier, longtime cortical modeler Stephen Grossberg calls this circuit for imagination “folded feedback.” If you close your eyes and imagine a hippopotamus, the visual area of your cortex will become active, just as it would if you actually were looking at a hippo. You see what you imagine.
.
.
.
.
