BEING ABOUT Chapter 9. Kantian stories
Spatial function and representationI therefore concluded that concrete spatial cognition ... is the grounding medium within which the quasi-spatial (symbolic) operations occur ... Luria 1973, 241 I am myself given to remarking that all knowledge is orientation. Grene 1995, 108 I have been suggesting that IPL areas that now are asymmetrically lateralized for representational specializations have their evolutionary and developmental origins in symmetrical structures facilitating various forms of basic spatial aboutness. Through the course of lateralization, counterpart structures that are similar in form and function will be altered so both form and function are different in the end. Structure necessary to basic spatial functions may, however, be only partially rebuilt. An example might be the left hemisphere organization of pointing gestures used to make syntactic distinctions in ASL. Bellugi describes deaf babies as beginning with nonsyntactic pointing, the way hearing babies do. As they begin to learn sign language in the year between one and two they switch from concrete referential pointing to syntactic pointing (Poizner, Klima and Bellugi 1987, 22). Right handed pointing presumably is organized through the left hemisphere from the first, so the right hand's shift to syntactic pointing will not be a hemisphere shift but a reorganization of pointing networks, with emerging differences in connectivity and connective consequence. If there are traces of original structure in representationally specialized areas, there will likely be traces of original function too. If this is the case, we might be able to notice a remnant spatial character in various kinds of representational function. This chapter describes evidence that basic level spatial function is central to several of the highest of high cultural uses of representation. Electroacoustic perspective'Country', as a word, is derived from contra (against, opposite) and has the original sense of land spread over against the observer. Williams 1973, 307 Electroacoustic music, like any representational medium, sets up in listeners a mix of perceiving and simulating; the listener is really hearing -- there are actual sources of pressure waves, loudspeakers in chosen positions in a room -- but hearing is also being used simulationally, so the listener is seeming to hear a source at the horizon, a swarm of sources hovering, a something arcing overhead. There are natural auditory illusions: wave reflections off a wall can make us mislocate a source, the way a mirror can. A recording that makes us seem to hear a bird in a meadow when we are really sitting in a room sets up illusion by reproducing acoustic conditions as they existed in the real meadow, the way a photograph reproduces an optic array. But computer music composers can design illusion by synthesizing acoustic structure directly. Transfer functions -- mathematical descriptions of the frequency, phase and amplitude covariances specifying source location -- include IIDs, interaural intensity differences, and ITDs, interaural time differences. Head-related transfer functions, HRTFs, include mathematical descriptions of wave interaction with torso, head, pinnae, or ear canals. Composers use these and many other functions to set up an acoustic array whose structure, delivered through speakers, is convolved with the existing acoustics of the room. A listener sitting too near a speaker will hear the actual sound source; a listener positioned optimally will seem to hear another space altogether: I was in a black space of transparent planes. Something happened to the atmosphere as if its grain were being polished. There were tissues moving at depths, not ethereal, transparent but strong, like sheets of rock seen by a god with x-ray eyes. Then that stretched thread of the sound of a human instrument, like brass, like a bagpipe, but an edge of the shred of the sound, drawn into a bright line, human concentration, vanished to a point on the horizon. I was so present in the space that I was wanting to turn my face to feel its air. Bolt upright on the edge of my seat, cracked from throat to navel, turning my face in an occult north I never wanted to leave. This audience description of a 16-channel diffusion of The Ghost of Eriboll (1994) by Peter Manning shows how the spatial simulation created by this composer is only partly aural in its effect. There is seeming to see blackness, brightness, transparency. There is also a sense of form and location that is spatial more than visual, or abstractly visual, perhaps -- transparent planes, depths, a taut line to a point on the horizon. There is somatic and act participation too. The listener is feeling herself to be really listening but is also seeming to feel herself pricking her ears as if to hear something far away passing out of earshot, just there, a little to the left of center, a location that, when she turns her head, is stable as a hill. When computer music evokes a panorama in this way, we can suspect a right hemisphere network, since many right hemisphere specializations are relevant to the experience. Right auditory cortex is responsive to environmental sound (see Chapter 8). Spatial working memory is right lateralized (Milner 1996). Parietal-frontal networks in the right hemisphere have bilateral control for both motor and sensory aspects of exploratory action. The right hippocampus is critical to an integrated sense of place. The right inferior parietal is important in object localization and configuration. Participation of right prefrontal mirror cells might account for act synaesthesias, sensations, for instance, of performing a motion as well as observing one. There are, moreover, sensorimotor populations in the right superior parietal that integrate locational perception, act organization, action for purposes of perception, act somatosense, and simulation in any of these modalities. Response in these act integration fields of the SPL seems both multimodal and sensorily thin enough to be involved in the kinds of spatial effect described above. Cross-modal visuality evoked by spatial music may be collateral to the somatics of neck, eye and ear tensions present when we orient to a location, and it may also be touched off in occipital cortex by auditory activity. Primary visual cortex in congenitally blind adults shows marked metabolic activity to auditory and tactile stimuli (Wanet-Defalque et al 1988). When subjects imagine a song, parts of visual cortex light up (Zatorre in Blakeslee 1995, C10), and when subjects are asked to imagine the sounds made by familiar objects, activity in visual areas is greater than when subjects are asked simply to imagine seeing the objects (Goldenberg 1991, 618). We can hear around corners and in the dark, and this may be an evolutionary incentive to tie audition to visual simulation; we hear an animal, and hearing it primes us to imagine seeing it. Imagining that we are seeing it is part of knowing what kind of animal it is. Auditory localization is calibrated to eye position at many points, both in the midbrain and in the cortex. The tie between audition and visual priming may be so compelling that it leads us to understand sound in inherently visual terms. Halpern and Zatorre (1999, 703) report that remembering tunes involves a forebrain area near the frontal eye fields, suggesting a close relationship between musical memory and visual motor memory. Rao et al (1997) find this area active in pitch discrimination too, as if an aspect of eye motion is an intrinsic part of musical attention. Visual, muscular and somatic involvement in audition are also notable in ways we think about the sort of hearing we are engaged in at a concert of the kind described above. We are not hearing sounding objects, the way we hear a dog sniff or wind ruffling the leaves: we are hearing abstractly, the way we see abstractly when we look at a nonfigurative painting. And yet, as we are seeming to hear -- and perhaps also seeming to see and to feel ourselves seeing -- locations that are not locations of objects, there nonetheless comes to be a sense of some sort of form or object at the locations we are hearing. The sense of a heard object, 'a sound', is so deeply a part of the way we hear, that we cannot speak about audition without it; but it is in an important sense metaphoric. There is no thing at the location implied in the way we are hearing, and yet we can often describe a shape, a texture, a size, and sometimes a color. The sense of watching something move can be so compelling it does not occur to us that we are imagining as well as hearing. Computationally composed music organizes us to be about things we are not with, always, of course, by means of something we are with. What is the mix of perceiving and imagining particular to the experience? What is it we are about in the presence of this music? What is it we are with, and what is it we are seeming to be with? The most salient and persistent simulational fact is the fact of orientation. We seem to be about locations. We experience ourselves as oriented toward sonic objects by axes that have our here tightly reciprocal to a located there. We have, in fact, organism perspective, which is in this instance acoustic perspective, which may also, not incidentally, be visual perspective, somatosensory perspective, and act perspective. We are tensed to perceive something at that location, and to some degree we are ready to act toward that location. We have a simulational task axis that mutually specifies our own and an object's position. The illusory sense of location is built around and within an actual sense of location. With a sense of hearing located things there is a sense of being in a space that is and isn't the space of the hall. The drifting transparent planes could be in the air of the hall, but that bright small something fading over the horizon can only be miles away. So actual and imagined spaces are only partly incommensurable; the touch of actual air on the skin of the face, the muscular tension of the ear tuning itself to frequency characteristics of actual wave trains, and even the ordinary, homey presence of postural self-sensing, can all participate in supporting illusory aspects of the experience. Perceptual and simulational aspects are ontologically distinct, but they occur by means of an integrated net and we need not distinguish them in the event. Denis Smalley describes his work as creating "an impression of space" through stability and continuity in time: For example, ongoing, sustained spectral textures showing very slow or no signs of longer-term evolution can be perceived spatially if the listener becomes less aware of forward motion and the slower levels of time passing. In other words, continuing existence can approach a quasi-permanence analogous to the contemplation of the visual permanence of a landscape. Smalley 1992, 534 The avant garde musician Brian Eno also speaks about his work in explicitly visual/tactile/spatial terms. He describes the process of composition as coming to know a place: There is a point in making a piece where I suddenly get a sense of where I am -- I can begin to sense the geography, the light and the climate. I was really moving in to a kind of landscape sensibility of music. There are foreground events, events not so close to the earth, ones that become misty and indistinct and then occasionally a hint of something out of earshot. I like this idea of a field of sound that extends beyond our senses. Eno 1996 Like Smalley, Eno also talks about sonic texture, which he calls the one innovation that really characterizes the recent period in music, and sonic shape. Eno describes one of his own pieces, for instance, as a "beautiful random mixture of things like the raindrops, with little flurries of things within it like icebergs". Pictorial perspectiveUnderstanding that we use actual space to support simulated perception and action can also help make sense of our uses of pictorial representation. Gibson's crucial new move (1970) was to understand that pictorial media have been developed to structure light so it will set up simulational structure in a viewer. Concretely, a picture is always a physical surface ... which either reflects light or transmits it. It is an object, in short, commonly a flat rectangular one, but what is unique about it is the light coming from it. The surface has been treated or processed or acted upon in such a way that the light causes a perception of something other than the surface itself. If an eye is actually stationed in front of the picture ... there has occurred a perception at second hand -- a vicarious acquaintance with an absent scene. Gibson 1982, 263 When we stand in front of a tree we see it by means of bodily response to the structure of the light reflected by the tree; when we look at a picture of the tree we will -- in some very partial way -- be seeming to see it, also by means of bodily response ordered in ways relevant to the tree. This is most obvious with natural images like reflections. When we see a tree reflected in a puddle, the light structured by the tree is reflected in so uniform a way that it reaches the viewer with its original structure undisturbed. We can seem to see the tree in sharp detail. There is a sense in which we could say we are actually seeing the tree rather than seeming to see it, but we are seeing it indirectly -- by redirected light. (This is also the sense in which we actually see ourselves in a mirror.) The seeming in this instance is partial -- it belongs more to the where than the what. Perhaps we could say part of the visual system is seeing, and part of the visual system is seeming to see -- but only in the sense that it is misattributing spatial location. It does so, presumably, by using something in the structure of the reflected light that was contributed by the puddle rather than the tree -- namely the direction of the light. Photographic technologies have been developed to focus and record just those structures in reflected light that will evoke the responsive organization by which we can seem to see. There is an ease about our use of photographs that lets us disregard the technical labour that has been necessary -- the shaping of lenses that permit us to pick out of ambient light just the structure contributed by some particular object, and the deployment of photosensitive materials that will maintain or enhance luminance ratios. A similar ease about our use of drawings and paintings lets us disregard the millennia of technical experiment that allow us to make marks whose overall organization will reflect light to fictive effect. The technical experiment has been experiment with materials, but more importantly it has been cognitive experiment; we have had to discover the visual system's weaknesses, along with the devices we can use to exploit them.
It has often been noticed that early civilizations, like young children, begin their experience of pictures and models with representations of single objects -- one bison, one human figure. How do we use a toy or figurine? Think of a clay torso 25,000 years old, a plastic horse or car, a doll baby. What do we perceive, and what do we imagine by means of what we perceive? We are not confused: we see and touch, and feel ourselves seeing and touching, the actual object, a terra cotta or a plastic form. And yet the sense of recognizing another kind of object is so convincing we don't hesitate to give the toy or model the name of the thing it represents: 'a torso', 'a car'. Something about our relation to the surface organization of a three dimensional model is causing us to seem to see the thing: we simulate very partial but very confident object recognition. For a child playing with a toy, much more is involved than recognition: along with seeming to see a truck or a baby, the child seems to see it moving or acting. This sort of play usually includes speech, which organizes further simulation so that whole circumstances are imagined. Part of the point of the play is what can be felt, excitement about the racing car, tenderness toward the crying baby. In all of this, many things about the perceiving moment are being ignored -- the feel of plastic, the smallness of the toy, the gestures that move the car along the ground. But, at the same time, the basic perceptual attitude, a gaze-based orientation to a something located at some particular place, is common to actual and imagined circumstance. An actual task axis is being used to organize simulation. More complicated acts can be built around this motor core: the doll baby can be kissed, patted and thumped. More schematic, but still amazingly effective in simulational terms, are line drawings, which may be very simple object profiles drawn in black on white. What we have in front of us when we look at such a drawing is an organization of oriented discontinuities, luminance contrasts set out on the two dimensions of a sheet of paper. Given this visual minimum, we recognize, we seem to see, a hammer, maybe, seen from some particular angle, though not at any particular place. Given this seeming to see, we may be ready to come up with the hammer's name, or to remember its balance in the hand. Like looking at a toy, looking at a line drawing requires actual orientation, an actual gaze axis. As with looking at a toy, some of our actual acting in order to perceive will be simulationally irrelevant: we are focusing at the distance of the page, not the distance of a hammer. There is no binocular disparity of the sort that gives us three-dimensional vision, but we are seeming to see something with a three dimensional shape. Deregowski makes the connection between form and outline by way of what he calls typical contour: Whenever a child is required to draw a picture from a bulky model, the drawing made tends to reflect the typical contours of the surface of the model. A typical contour has been defined as a line on the surface of a solid which connects the points of greatest change in curvature ... subjects find discrimination among models having more pronounced typical contours markedly easier than discrimination amongst models having less pronounced typical contours. Deregowski and McGeorge 1998, 35-36 What is common to our use of both figurines and line drawings is object recognition reliable across differences of size, color, position, and orientation. Object identifying, classifying, and mnemonic evocation on the basis of typical contour seems to be a function of areas in the ventral stream, neuronal columns in TEO and TE where object constant columns will respond to profiles if they not atypical; neurons of the inferotemporal cortex in anesthetized monkeys will in fact respond to simple line drawings of objects almost as strongly as to the sight of real objects (Tanaka 1993, Sakata et al 1996, 251).
Seeing a line drawing of a hammer, we seem to see an object: we seem to see it in a space that is not the space of the page, and we seem to be facing it from some particular direction. What is increased with a perspective drawing is the latter sense that we are looking, the sense of our own oriented participation in seeing. When we add a horizon line to the drawing, we add an illusion of the object's placement in an enclosing environment. We also add a sense of being ourselves oriented to a background as well as to the object. When we animate this drawing, there is an illusion of motion either of an object relative to viewer and environment, or of the viewer in relation to object and environment. Like any representational object, these drawings use perception to evoke simulation. The simulation evoked is very partial and yet very robust, given the economy of its perceptual means. We seem to see the object in a manner that is both exact and abstract. It is abstract in the sense that, if we seem to see the object's surface materiality (color, texture, opacity) or even its size, we do so ad lib, without instruction from the drawing. It is exact in the sense that the placement of corners, edges, and surfaces, as interrelated in the object, and relative to us, is entirely specific. Backgrounds can also be evoked with an abstract but exact specificity of this sort. Line perspective drawings can be thought of as outline drawings of whole environments. As with object outlines, there is nothing in the drawing to evoke materials, lighting (source direction, intensity, color) or atmosphere. What is specified by a perspective drawing includes the viewer's position. Surface orientation and viewer orientation are mutually dependent: visible surface orientation is orientation to the viewer. "The position of 'here' can always be made out, i.e., the point at which the observer is standing. This point is not actually in the picture, but it is in the space represented by the picture" (Gibson 1982, 240). 'The space' evoked by the perspective drawing is thus a simulational experience of mutual relations among viewer, located objects with their located parts, horizon, and ground.
Every representational medium organizes a responsive net enacting some mix of perception and simulation characteristic of that medium. What can we know and suspect about the mixture characteristic of perspective graphics? We use perspective drawings by means of a non-simulational sensory-motor base of actual gaze direction, saccades and fixations, focus at the surface of the drawing, and stereoscopic fusion at that distance. Unlike the experience of a computer music panorama, almost everything about the actual perceptual axis is incommensurable with the simulational axis suggested. The space in which we look down at a drawing and turn the page is clearly incommensurable with the space in which we seem to see a Greek temple against a distant horizon. When we look down at a page and seem to be looking toward a distant horizon, focus, fusion, vestibular sensation and head position must in fact be ignored, overridden. This is even more striking with the wireframe animation that makes us seem to see a moving object. When we are actually tracking a moving object, we feel the muscles accomplishing vergence and lens accommodation. Watching a wireframe animation, the stability of vergence and lens accommodation will be inconsistent with seeing motion. With the computer music panorama described above, the actual source of sound was not heard as such. But a perspective drawing, as such, is not invisible. We are also about the drawing as a marked surface. There may be relative invisibilities, the lines may be more seen than the paper, but we do see black lines on white paper, and we see them where they are, at their actual distance from us. The lines are seen as lines, but we also seem to see them as object edges. Why is there no difficulty about this? The only aspects of actual somatic participation that are not inconsistent with what is being simulated may be just the muscular participation of seeing at all, a faintly palpable I here now of somatic presence. It must be that, for the rest, simulational vision and perceptual vision are simultaneous but segregated.
Line perspective, like acoustic perspective, seems to be a right parietal specialization. Recall that ability with drawings is itself a right inferior parietal specialization, that an inability to name or recognize drawn objects results from right IPL damage, and that von Stein et al found a right hemisphere increase in temporal-parietal synchronization when subjects identified objects in a simple line drawing (von Stein et al 1999). Right IPL lesions damage the ability to make and understand diagrams, maps and plans, and in general to imagine three-dimensional facts in response to two-dimensional representations. In mental rotatation tasks, which are based on perspective drawings of arrangements of blocks, subjects are asked to imagine these arrangements from different angles, or to remember an orientation. Imaging studies of these tasks show increased activity in the right hemisphere intraparietal sulcus. In many subjects this activity also extended into both the angular and the supramarginal gyrus, and into the occipital-temporal area associated with motion perception (Cohen et al 1996). A picture subtends a limited part of our visual field -- about 45-50 degrees solid angle -- and it is seen with central resolution, focal and foveal. It is seen, that is, with an area of the retina served by both magnocellular and parvocellular structures, so activity may be propagated along both ventral and dorsal streams. A dorsal network will be organizing muscular action needed for perception of the picture as such. If we recognize and name some particular kind of object, we can suspect involvement of ventral structures responsive to typical contour. von Stein's findings suggest some sort of right hemisphere temporal-parietal synchronization will be present with any use of pictures of objects. This much will be true for any outline drawing. What is the particular contribution of organized perspective, with its parallel and convergent structures? And what more will be added if the perspective drawing is a wireframe animation, say of a cube stationary in relation to a ground surface, with observer viewpoint seeming to move? A black and white line drawing seems to make little use of the spatial and frequency resolution given by the parvocellular subsystem. People who lose parvocellular function are able to draw and to copy drawings, but people who lose magnocellular function are not. Moreover, line convergence used to indicate perspective fails under equiluminance (Livingstone and Hubel 1988). Because the magno-based dorsal stream is insensitive to frequency differences but very sensitive to broadband luminance differences, these results suggest that a dorsal subsystem is critical to linear perspective effects. Further, broadband cells in area MT in the dorsal stream have been found to respond to line convergence. Motion perspective "seems also to depend on luminance contrast and could well be a function of the magno system" (Livingstone and Hubel 1988, 746). We have seen that the dorsal stream works with a segregable, frequency-insensitive structural visuality based on edges and corners -- a sense of an object as an oriented volume, which is a sense of the object related to the requirements of reach, grasp and other act subsystems. In Gibson's terms, a perspective drawing deals with objects in terms of surfaces rather than marks on surfaces -- just the way the dorsal stream deals with them. Pictorial perspective thus seems to need a particular kind of relation of ventral and dorsal systems. It might be fair to say outline drawings are seen with the ventral system but when a perspective notation is added, they must be understood with the dorsal. We have seen how, even in basic perceptual situations, different matrices can be responding to different things, or even to different aspects of the same thing. Could we say some matrices in the ventral system, in V4 maybe, participate in seeing a two-dimensional drawing, while others participate in seeming to see a three-dimensional object, and that some matrices in the IPL and SPL are organizing stationary attention to a locus eighteen inches from the eyes, while other dorsal and medial matrices are seeming to organize attention to a moving locus that could be a hundred meters from the eyes? -- All of these foci, perceptual and simulational, being integrated in a core dynamic subnet that may also include speech and other areas. We can easily imagine the same stimulus conditions, and the same early visual response, as being used in different ways by different subnets in order to effect the many nuances of simultaneous perception and simulation. What we gain with a wide net understanding of representational function is the beginning of a way to notice the curious cognitive segregations and integrations we have taken for granted.
... all abstract phase-, state-, and Hilbert-space entities, seem to exist in a geography, a space, borrowed from, but not identical with, the space of the piece of paper or computer screen on which we see them. All have a reality that is no mere picture of a natural, phenomenal world, and all display a physics, as it were, from elsewhere. Benedikt 1991, 21 Picturing skills developed during the Renaissance involved elaborations of both object illusion and location illusion. The light reflected by a Renaissance painting causes us to seem to see not only identifiable objects and their groupings, but also, by texture gradients, illumination and atmospheric effects together with systematically applied perspective convergence, the background place surrounding them. Renaissance painters developed chiaroscuro effects -- illusions of object three-dimensionality created by highlights and shadows -- along with perspective effects because the two are intimately related. Summers (1990) has pointed out that a scene painted with consistent chiaroscuro will imply the position of light sources as well as the position of the viewer relative to both objects and light sources. Consistent chiaroscuro will thus help to organize seeming to see a particular scene from a particular viewpoint. If the scene is daylit, it will also organize seeming to see at a particular moment, since the sun is always in motion. Consistent chiaroscuro thus cannot be simply transcribed: in the time it takes to paint the smallest area, the light source will have shifted. So Renaissance chiaroscuro required explicit knowledge of perspective projection. Edgerton (1991) has traced the development of Giotto's viewpoint perspective to his contact with Robert Grosseteste's (1163-1253) geometrical optics. By using converging lines in space to set up seeming perception of a spatiotemporally unified space, Renaissance painters gave themselves a locational framework for other visual effects, such as shape-from-shading and detailed color. This framework was first drawn onto the canvas as a system of high-contrast lines. In the process of learning to organize paintings to complex fictional effect, painters had also discovered that pictorial representation used with standard measuring practices can become a notation for constructive procedures. Architects could use scale drawings where formerly they used scale models or took measurements from modules drawn onto the floor. Linear perspective drawings enabled the artists of the time also to be the engineers of the time -- specialists in noticing and imagining and thinking spatial relations in isolation from other features. The first generation to grow up with printed illustrated books was also the first generation to grow up with copperplate perspective drawings of scientific subjects (Edgerton 1991). That generation included Brahe, Bacon, Kepler, Galileo, and Harvey. A hundred years later Newton was skilled enough in segregated spatial imagining to be able to think his local real space extended indefinitely on all sides, part of an integrated absolute space containing large and small motions describable by differential equations. But before Newton there had to be Descartes, who understood that abstract location -- geometrical space -- if it is imagined paced out, measured, marked by imagined lines into Cartesian coordinates, can be used in conjunction with another representational medium, namely algebraic notation. We have seen that the right hemisphere is important to basic sorts of physical understanding: people with right hemisphere weaknesses can lack physical concepts such as conservation, seriation, multiple classification, space, number, substance, quantity, and weight (Bihrle et al 1989). Giftedness in spatial reasoning follows right IPL development. At the same time, mathematical and other uses of notation systems make preferential use of the left hemisphere. It is likely, then, that Descartes' innovation was a conjoined use of right hemisphere pictorial/spatial ability with left hemisphere notational speed, and that it was this bilateral network, rather than pictorial perspective alone, that enabled the science that followed. Linguistic perspectiveIn Chapter 6 I suggested at length that spoken and written language can be understood as the use of acoustic and graphic form to direct action and simulation, both in others and in ourselves. Language is able to effect maximum cognitive structure with minimal physically present form by triggering structure pre-organized through intensive training. Training with names sets up rapid left-hemisphere evocation of networks accessed through convergence/divergence zones in many cortical areas. Basic level aboutness seems in general to be evoked by open-class lexical means. Closed-class syntactic resources of a language may in general direct procedural effects. Cognitive mode or style is organized by many means, including rhythmic and sonic effects considered pragmatic rather than grammatic. The left hemisphere is linguistically specialized for the perception and production of acoustic and graphic elements of a language, but other linguistic effects, including text-level integration, involve the right hemisphere, so that in normal function language requires bilateral integration of a cross-callosal net. In the first two sections of this chapter I describe spatial effects as they are organized by and experienced in musical and graphic media. A novelist may set up similar effects by linguistic means: From here, black desolation down there at the river is before his eyes again. They are tramping past him on the road in their usual weekend peregrinations. He hears them at his back and they hesitate to overtake him, it's as if he's leading them in procession, ridiculously, for a few moments, and then they surround him at a polite distance briefly while gaining on him, two men on the one side, and one of their women on the other. Gordimer 1975, 109 Nadine Gordimer's devices include deictic phrases (from here ... down there), prepositional phrases (on the one side ... on the other; at a polite distance; at his back; before his eyes), verbs with a spatial character (tramping, overtake, surround, leading), and nouns with spatial implications (peregrinations, procession). What is perceived when we read this passage, and what imagined? As is common with language function, we are not very aware of perceiving what we have to have perceived in order to imagine as we do, namely the linguistic forms -- the letters, the words, and the sentences. What we are directed to imagine by means of these more-or-less unconsciously perceived forms is a scene, a road at some distance from, and on a rise above, a river, and four people socially and spatially related within it. Viewpoint may be felt to shift. We begin by seeming to see from the point of view of a man being overtaken on the road. We are seeing from his here downward to his there at the river. We seem to see a river at some distance or other, but we also seem to feel ourselves looking downward toward it. We feel a reach of space between our place on the road and that place down there. At the same time we hear tramping behind us, and then hear it overtaking us. For the rest of the sentence, at he's leading them in procession ... and then they surround him ... while gaining on him, I find myself imagining the whole group from a point behind them. This viewpoint shift is not necessarily intended; it may occur because the author shifted viewpoint as she wrote, or it may be idiosyncratic to my reading. What is imagined normally exceeds instructions: I seem to see the river at the distance of a quarter mile, toward the left, at a very shallow downward angle. Other readers may see it at the bottom of a gorge on the right. My reading is influenced by earlier passages, but is only very loosely constrained by these paragraphs. We are directed to imagine a viewpoint; the way we imagine it is up to us, not coerced as it can be by musical or graphic means. The point is, however, that language is like other representational media in evoking spatial simulation that includes somatic, viewpoint, and act perspective.
Deictic elements in spoken natural languages are forms that have been thought linguistically anomalous because they cannot be understood outside a particular context shared by speaker and addressee. Here, that, I and you are examples of deixis in English. Where speech is used to organize perception and action among people present together in a shared space, its construction and comprehension is constrained and completed by mutual perception of that shared space. Deictic forms are like various nonverbal ways of using shared context during speech. If there are two Mary's in a room, we will look at the one we mean. When we and our addressee are working at a common task we can be cryptic because we are seeing the same things: Not so near can mean Don't put the rock there; move it further from the other one. Posture, gesture, gaze and context effects such as these are assigned to discourse pragmatics rather than grammar, but, when they coordinate mutual attention and organize our construal of present situations, many linguistic functions thought to be syntactic rather than pragmatic can also be understood as forms of presence deixis. The relation of deixis to referential point and gaze is suggested by the derivation of the term from the Greek deiktikos: able to show. What I am calling presence deixis corresponds to the first of three kinds of deixis distinguished by Buhler in 1934, deixis ad oculos. Pierce includes this sort of deixis, with pointing gestures, among indexical uses of signs (1980). Deictic forms in English include intransitive prepositions like upstairs, time expressions like tomorrow, and motion verbs like to bring and to arrive which must be understood as anchored to a deictic center. Since all of these expressions are also understood when we are not speaking about a shared physical context, it is plain that deictic expressions will also work for mutually imagined circumstances. In my sense of it they are not unusual in this, since most aspects of a language require and organize mutual simulational aboutness. In their simulational uses deictic expressions can in fact be seen as central instances of linguistic function, instances in which the social management of simulational aboutness is seen with particular vividness. Buhler calls simulational uses of deictic forms deixis ad phantasma, and suggests that somatic experience deriving from actual perceptual presence -- Korpertastbilder -- acts as a sensorimotor core, an I here now, animating simulational experience evoked by their means. Simulational deixis is also called discourse or narrational deixis, since it can establish comprehensive fictional orientation of the sort evoked by Gordimer's passage. A simulational orientation or discourse space may be sustained through very long texts or it may be dropped after a sentence. It may or may not be a character viewpoint: ...it's also possible to choose a reference point -- a place with which the narrator somehow associates himself and his reader in imagination -- which has no particular association with a central character. Thus, if I'm talking about an uninhabited island in a little-known lake in Minnesota, I can talk about a loon 'coming' there at night and about the waves 'bringing' things to its shores. Fillmore 1975, 67 Since signed languages explicitly use space for grammatical functions as well as for what are considered pragmatic functions, studies of signing can also make the continuities between deictic and non-deictic uses of space particularly obvious. In a recent collection of studies of the uses of space in sign languages (Emmory and Reilly 1995), Scott Liddell shows how presence deixis and simulational deixis may coincide, the way perceived and imagined space can coexist when we listen to computer music. Linguistic action in American Sign Language is largely gestural but it also includes syntactic use of eye gaze, facial expression, head position and movement, and body position and movement (Liddell 1980). Along with the contour of a sign, its spatial extent and the number of repetitions, the location of a gesture has syntactic import. Location matters in two ways: it matters where you produce the sign, and it matters where you point it. Pronouns in ASL consist of a root sign and a pointing gesture. Adjectives will be pointed toward the object or person described. There is also a class of verbs called indicating verbs, which consist of a verb root and an indicating gesture: the verb form may be pointed toward a referent, or it may move from one referent to another. Verbs which take an object may for instance be signed moving from the subject toward the object of the action: he-there FLIRTS with her-there. And certain verbs are always signed toward specific parts of the addressee's body: THINK toward the head, GIVE toward the chest. Liddell talks about three kinds of signing space: real space, surrogate space and token space. I will modify his terminology slightly, since all three of his categories are using real space, that is, the space mutual to speaker and addressee. The important contrast is about kinds of use made of that space. Liddell's surrogate space is a use of space structurally very similar to perceptual use of mutual location; the only differences are consequences of the fact that real, mutual space around the conversation is being used to support simulation rather than perception. Kids using actual living room furniture to play school would be using space in a surrogate way. Signers use space in this way when they are talking about something that happened last week, or reporting speech, or sometimes when they are constructing conditionals. For signing purposes people and objects that are not present are imagined present, full size, and at realistic distances from the signer. Talking about Mary, who is not present, the signer establishes a locus to which future pronouns referring to Mary can be directed; this locus is treated as if it is Mary's height. If the signer wants to say Mary gives something, the GIVE sign will be directed toward a locus the height of the actual Mary's chest. To say Mary flirted with Paul, the signer will direct a FLIRT sign from the Mary locus to Paul, if he is present, or to a Paul locus if he is not. Referential shift is a use of surrogate space to report speech, among other things. To make it clear that she herself is not the intended speaker, the signer can step into a different spot and sign from there, establishing a character viewpoint. Referential shift can also be indicated by shifts of head position, torso position, facial expression or gaze. Surrogate space is plainly still a deictic use of present space to support mutually coordinated simulation. Liddell's innovation in ASL studies is his description of token space as similarly deictic. Token space is sign subspace -- the space in which signs are performed -- used as if it were compressed surrogate space. The signer will still establish a locus for imagined Mary, but will establish it by a small stroke of the pointing finger dropped under the sign for Mary's name. That locus will then go on functioning as an index for pronoun co-reference, adjectives and indicating verbs. Directed verbs such as GIVE will be pointed toward relative heights at that locus as if a token figure is being imagined (Liddell 1995). This sort of reduced, token use of space can be like full size surrogate use of space in supporting conversations about spatial facts: Mary-there KICKED Paul-there. But it can also be used to talk about non-spatial facts and relations while retaining the structures of discourse about spatial facts. Token space can for instance be used metaphorically to talk about time: the present-here, the future-here, and a trajectory from here to here. A signer can even use this spatial understanding of time in a meta-pragmatic way, to orient an audience within the time course of the narration itself. The beginning of the story, here, the end of the story here. Discourse cohesion can then be maintained by signing or gazing toward time-points indexing events of the story. The signing hand may be positioned at the point the story teller has reached, while the signer holds her gaze on that hand. Mcneill and Pedelty (1995) point out that hearing speakers use gesture and gaze in similar ways, looking to the side when they are making back-references, marking topic changes with hand beats and looking directly at the addressee when they want to indicate an aside. The use of token space that is most suggestive in general cognitive terms is a sort of logical and/or attitudinal use. When comparing or contrasting two notions or two alternative conditions, the signer will as if establish two token spaces, one on the right, one on the left. A conversation about art and science, for instance, can be set up by placing motions establishing loci, art-here and science-here, where pronouns, adjectives and verbs may later be indexed to them. Both will have their localized nominals, with internal co-reference of the kinds described. The signer will thus be able to maintain whole alternative contexts, looking from one to the other, constructing one in contradistinction to the other. The signer can also enact a range of attitudes toward those juxtaposed wholes -- ignoring one and concentrating on the other, drawing the two together or separating them further, setting one aside. The process of thinking about alternatives is thus being supported by actions -- eye motion, body orientation, gesture -- which are actual rather than imagined, but which are actions toward, or involving, imagined entities. It may be that ASL allows us to see people thinking in act-metaphoric ways also used -- covertly -- when they are alone or speaking orally. McNeill and Pedelty, describing gesture use by hearing people, note that abstract metaphoric uses of space may actually be built around earlier more concrete uses within the same discourse. In one of their examples a narrator gestures in a certain direction when describing a location where a film character exits a scene. Immediately afterward a gesture in the same direction is used to talk about the beginning of the next scene. That is, the locus used to support imagining a character's departure is used metaphorically to indicate the starting point for a new scene. ('Starting point' -- we do it in English, too.) Both are token-scale uses of the actual space surrounding the conversation, but the first supports spatial simulation and the second is act-metaphoric. "The gestures looked morphologically identical, but their semantic value was different with each occurrence" (McNeill and Pedelty 1995). A continuum similar to the continuum Liddell noticed for ASL use of sign space may be noticed for nonlinguistic media as well. In graphic media, a theatrical backdrop or trompe l'oeil painting would be a surrogate use of deixis. An small outline drawing instead uses compressed, token space to think about -- to seem to see -- an absent object. A more complex use of token space would be a map or diagram used to direct spatial action. More complex again, but still deictic in the sense that it makes essential use of sensorimotor orientation, there is the metaphoric employment of pictorial space to support talk or thought about non-spatial things. An example would be a graph using the vertical and horizontal dimensions of a sheet of paper to indicate correlated variation of two quantities. Aspects of what are taken to be ordinary syntactic uses of space in ASL can sometimes also be seen as a kind of deixis. ASL indicates clausal relations by establishing clause nodes at particular loci in sign space and then placing signs under, or to the right or to the left of that node. This use of loci is similar to alternative conditional spaces or to loci marking positions in story structure. Liddell's paper is controversial in ASL studies because, by finding a continuum between clearly pragmatic and clearly syntactic functions in a language, he unmakes what has been a founding dichotomy in linguistics. What seems likely is that we will discover that many properly syntactic functions are also based on simulational action or imagined motion perception, and that much grammatical structuring is performed by means of compressed, left hemisphere sketch-remnants of simulational spatial behavior.
Do not words excite feelings of Touch (tactual ideas ) more than distinct visual ideas ... the Question is of great Importance, as a general application -- Coleridge Notebooks II, 2152 There is a subtle but distinct difference in semantic effect between The road lies between the mountains and the river and The road runs between the mountains and the river. Leonard Talmy identifies the difference of feel between the two verbs as a difference in fictive motion (1996, 268), registered through what he calls an abstract sensing. Here is another example: The road ran into the distance. Reading this phrase, it is as if we are looking rapidly from somewhere near by, the source, along a trajectory to a goal, the distance. The muscular feel of this simulational sequence is not additional to but part of our comprehension of the sentence. The linguistic effect is fictive, in addition to being simulational, in the sense that we are seeming to see motion in relation to something we are also and simultaneously imagining as motionless: "fictive motion is coupled generally with factive stationariness" (213). Talmy compares the effect to the paradoxic visuality experienced with motion after-effects. When we have been walking in fog and stop suddenly, the landscape can seem to be rushing away at the same time as it can be seen to be standing still. Similar effects can occur as collateral imagining: seeing a coyote's tracks in snow, we may seem to see the coyote running; seeing brush strokes on the side of a bowl, we may seem to see the motion of a brush. Other instances of sensorimotor fictivity evoked by verbs include: 1) frame-relative motion, The fence descends; 2) pattern paths, The trail of drips across the floor; 3) access paths, The vacuum cleaner is down behind the hamper; and 4) coverage paths, The ranch spread over the plateau. Like viewpoint in the Gordimer passage, fictive motion effects are underspecified. Individuals may experience the same example differently, and the same individual may deal with the same example differently on different occasions. What is felt as moving, whether it is the object, the self, or just the direction of the gaze, varies in inscrutable ways, but "every speaker experiences a sense of fictive motion for some fictive motion constructions" (Talmy 1996, 215). Talmy describes fictive motion as one of several kinds of structure-sensing normally present with other kinds of perception but inherently multimodal. It can be part of vision, audition, somatosense, and touch. In any of these modalities it is "a faintly palpable level of perception" (Talmy 1996, 248), not exactly vision and not exactly somatic sense, sometimes with a dimly felt sense of action for purposes of perception -- eye motion, maybe. Thought of as visuality, it has to be considered abstract. Talmy compares it to the oddly nonvisual visuality experienced in form completion illusions; when we experience a pac man as a circle with a bite taken out of it, we sort of see, or is it feel, the missing part of the circle as a fictive presence. We have come across this structural visuality before, with the intersecting planes evoked when we listen to music. We have also encountered it with the strangely non-visible three-dimensionality evoked with line perspective drawings. This material suggests a structural or spatial vision participating in sentient vision in an effective but barely visual form, as if ascribed to what is actually seen -- a colorless and transparent sort of grey vision. It is felt as a sort of vision but it is felt as a kind of action too, or as somehow more closely tied to action and motion than to the sense of scene and objects.
Locational language is one of the few ways we can come to notice a structure subsystem of this sort, because fictive structure and motion effects can be evoked by language forms apart from the perceptual conditions in which structural perception normally occurs. In the midst of active vision there is too much other visual structure. Locationals direct us to see or imagine one object in a spatial relation with another. Open class locationals include many names of objects and places -- roof and street -- that evoke spatial relation schemas. They also include many sorts of implicitly relational verb: The train crossed the trestle. The same function is often performed by prepositions: The train chugged across the trestle. Prepositions are closed-class terms -- there are only 80 to 100 in English. A small subset of prepositions (they include upstairs and outward) is intransitive; most, like across, must be used with an appropriate nominal in the object position. Locational prepositions specify direction, relative distance, and various sorts of contact or containment. In all of these instances we are being directed to imagine a foreground object spatially related to a background object. Prepositions of motion may evoke positions of motion paths of one object relative to another (via, along, across, around, parallel to, in line with), or directions of motion of one object relative to another (to, toward, from, away from). We may be directed to locate a foreground object relative to a background object by reference to the intrinsic spatial organization of the background object (above, below, horizontal to, behind, in front of). Prepositions may also evoke a distribution of foreground object or substance relative to a background object or substance (all over, throughout, all along) (Landau and Jackendoff 1993, 235). Prepositional effect is characteristic of closed class linguistic effect in that it is procedural; when a preposition directs us to imagine two things spatially related in some specific way, it is also directing us to specific kinds of simulational participation -- it may be directing us to seem to look from one to the other and so maintain two axes related by eye movements, perhaps, or to seem to see one of the objects with parvocellular detail and the other with dorsal abstraction. In general, prepositions elicit a response that ignores details of reference object shape (Landau and Jackendoff 1993, 258). Compare red wheelbarrow in the snow to snow in the red wheelbarrow. If a noun is used as object of a preposition, as when it is used as object of a verb, the presence of the preposition directs us to imagine that object schematically, as a background object. At the same time, it directs us to imagine the subject of the preposition more fully. Choice of preposition may direct the way we see or seem to see the reference object in other ways as well. Fillmore (1975) gives these examples: at the corner and in the corner organize us to see the corner from different directions, while in the grass and on the grass make us see a different kind of grass. Landau and Jackendoff relate the schematic or abstract spatial imagining of objects evoked as reference objects to the sort of perception needed for purposes of action. Both Talmy (1996) and Landau and Jackendoff (1993), hypothesize that prepositional function shows a likeness to what we think is happening in the parietal. What intrigues us is that ... the location system in language has just about the right properties to interface with the "where" system described by neuroscience. We therefore conjecture (a weaker term than "claim") that what we find in the language of places has a fairly strong homology with the [tracking] of objects and places situated in the "where" system of the brain. Landau and Jackendoff 1993, 257 We have seen that, while temporal object vision matrices are responding in detailed ways to the thingness of things, parietal systems are tracking them spatially and in less sensory detail. Talmy concludes, from the linguistic evidence, that objects are taken as points and lines for the purposes of prepositions (1996, 261), and that this schematization is an effect of the dorsal system. In chapter 8 I considered what basic human IPL function might have been, before language and other representing practices had begun to change brain structure culturally. Clues from remnant right hemisphere function in humans suggest the IPL is important to at least two sorts of spatial function evoked by prepositions. The supramarginal gyrus seems to be involved in perceiving or imagining the corners, edges and surfaces of things for purposes of grasp, and the angular gyrus has something to do with a positional sense of objects in arrays. It seems likely that the dorsal spatial structure subsystem conjectured by Talmy and by Landau and Jackendoff is the IPL rather than SPL. Another possibility is that IPL areas are connecting language areas to eye motion areas in the SPL. We have seen that nouns and verbs evoke activity through convergence zones in different cortical areas. When sentences and longer texts evoke whole spatial scenes, there will have to be ways of tying these separate nodes together so they participate in dynamically integrated wholes. The inferior parietal, whose development is a precondition for every kind of representational function, seems to be well placed to link and integrate sentence effects, since it is between act constant areas in the SPL and object constant areas in the temporal lobe, and next to phoneme and morpheme perception and production areas in the Wernicke's-Broca's stream. Since prepositions tell us where to imagine parts of a scene relative to each other, and how to imagine our own spatial participation in the scene, we may discover the IPL is a convergence site for prepositions. We have seen that the right hemisphere dorsal stream is more involved in bilateral spatial action than the left, and that the left hemisphere dorsal stream is more involved in rapid, routinized fine muscle motion. Landau and Jackendoff suggest the linguistic system for location is "closely homologous to" spatial response by the left hemisphere where system (Landau and Jackendoff 1993, 261). Kosslyn describes right hemisphere spatial function as metrical and left hemisphere spatial response as conceptual. Kosslyn's distinction between conceptual and metrical forms of spatial response may be redescribed in terms of Liddell's distinction between token and surrogate uses of sign in ASL. Left hemisphere prepositional response may be a system of distinctions made at token scale, so that rapid habitual use of above/below, right/left, inside/outside, and the like may evoke minimal versions of structures that, if they were evoked on the right, would be part of presence or surrogate simulation. We have seen that the IPL may also be important in tying left hemisphere linguistic nets to counterpart spatial function areas in the right hemisphere. If so, widely connected surrogate scale spatial simulation organized from the right hemisphere might evoke left hemisphere minimal simulation, which in turn would evoke a spatial name. Halle Brown suggests that discrete categories of location are a product of the interface between the left and right hemisphere spatial systems, not of the interface between the spatial system and language (1993). Circuits making minimally connected prepositional distinctions on the left might under some circumstances also set up more widely connected right hemisphere circuits via callosal connections with counterpart structures. An additional point made by Landau and Jackendoff is that prepositions treat parts of objects as if they were objects in their own right: The language used to express the relations of parts to objects is identical to that used for configurations of independent objects ... we speak both of a nose on one's face and a fly on one's face. 261 With both sentences given above, face is vaguely imagined, but nose and fly are both imagined with focal vision. We regard the part as a separate object with its own location, while the object of which it is a part is treated schematically like any reference object of a preposition. Mathematics and act metaphorMathematics is not a representational medium so much as it is a multimedia context or representational nexus; a function may, after all, be specified or evoked in many media: graphs, tables, a natural language description, a formula, a program, a gesture, or a working model. Mathematics could also be called a culture, in the sense that it is developed and transmitted by groups of people with intensive training in specialised skills. It is a culture that constructs reliable mathematical behavior for many purposes and by many means. In this section I will speak about only a few aspects of representing practices in mathematical culture.
Notational systems are central to mathematical cognition. Mathematics is not generally an oral culture: even skilled practitioners need written notation to support complex chained procedures. As is the case with natural language, a written record can be used to reconfigure complex cognitive states interrupted for an afternoon or a year. Mathematicians also make heavy use of notation to store precomputed results, which may, alternatively, be memorized or built into the design of a tool like a slide rule or a computer (Hutchins 1995). We do not generally think of written notation as a pictorial medium, although, like pictures, it is a graphic medium (>Greek graphikos, of writing). Written symbols are different from pictures in a very particular way: most do not use a fictional structure of light. Their use of pictorial subspace may direct simulation, but it does so by other means. When we read alphabetic writing, music notation, or logical and mathematical formulae, we make rapid graphic distinctions, usually with contrasty, color-irrelevant marks, and these distinctions in turn direct behavior and/or simulation of other kinds. Music notation may direct performance. When we read aloud we voice a linguistic score. The action directed by written language is, however, more often covert, and it may involve speech sound only marginally. Mathematical notation, like music and natural language notation, also scores and is a record of overt or simulational action, but logical and arithmetical symbols are used to evoke simulation of much more circumscribed kinds. Briefly stated, arithmetical notation mostly evokes imagined action with tokens.
Denise Schmandt-Besserat's (1992) story of the development of cuneiform writing in the Near East shows how such a practice could begin. Archeological evidence has cuneiform well-established in Sumer by the 4th millennium BC. Schmandt-Besserat, a French archeologist, was interested in why cuneiform is not pictographic, as is the independently developed early writing of China and Mesoamerica. Her theory, published in 1978, based on evidence spanning 5000 years and drawn from sites throughout the Near East, is that the prehistory of our early writing is also the prehistory of mathematical notation. Count-pebbles, picked up and thrown into a heap or dropped into a jar, would be adequate for transient counts of the sorts of things nomads might need to count, but an agricultural society settled enough to allow vocational specialization and trade would need to count and to record counts of many more kinds of things, and would need to be able to specify object kinds. The solution in the 9th millennium BC, Schmandt-Besserat says, was to use clay tokens shaped like the item being counted. Five measures of wheat: five clay cones. Ten jars of oil: ten ovoids. The next development, so radical it seems to have needed four thousand years, was that commodity and count came to be specified separately. Kind tokens would be small models of for instance jars, animals, fruit, people. Count tokens would be generic -- a wedge shape related to the cones formerly used for measures of grain. The resulting count record would be a collection of one commodity token and a number of count tokens. Such collections would have to be kept together, and two archiving systems have been found: tokens could be strung on a piece of twine, or they could be sealed in hollow clay containers. Although we still speak of strings of symbols, it was the latter archiving technology that had large undeveloped potential: eventually someone saw that, if you used your tokens to impress the wet clay of the container before you sealed it, you would not have to break it open to know what was in it. So then we find clay envelopes containing one jar token and 10 generic count wedges externally imprinted with the shapes of a jar and 10 wedges. There were two more stages preceding clay tablets as they were used in developed cuneiform; in the first, clay envelopes were impressed with tokens but empty; in the second, tokens were impressed on curved clay sheets. In final developments, count tokens/stamps were replaced by tokens/stamps signifying the names of numbers, and kind tokens were replaced by reed-stylus inscriptions, a change enabling a larger ready lexicon of commodity names. The surface space of the now-flat tablet was also used in a systematic way; a mark in the bottom corner would be taken as naming the donor of the goods recorded in the central area of the tablet. Can we think of the count-token progression in terms of Liddell's series of simulational uses of space? If we are dropping a pebble into a jar for each sheep that passes through a gate, we are engaged in action in real space, but part of that action is the creation of a record. Later we'll look at the pebbles and imagine sheep. We'll ask a question in relation to pebbles which is really a question about sheep -- the question How many? Looking at pebbles in the absence of sheep we are using the jar and the area around it as a token subspace. Here are (as if) the sheep in their pen: I'll need these for my taxes, these for food, and this lot over here I'll trade tomorrow. Thinking is being supported by coordinated motions of eye and hand, grouping and separating actions on the scale of a table top being used to think about grouping and separating actions on a larger scale. When counters are shaped like the commodity being counted, grouping and separating tokens would call up some sort of integration of ventral object constancies and dorsal act constancies, like the integration of object recognition and schematic action evoked when a child plays with a toy. But when commodity tokens are differentiated from count tokens, there is a potential for thinking how, where and how many separately from what. We can move placeholders, generic somethings, in a token space, using the hand-eye motions we have used playing with toys or thinking about sheep or heaps of grain. ( -- Or we can use these motions for nonrepresentational purposes, as we do when we play games with counters on a marked token subspace.) When we keep our counters enclosed in a clay envelope we can still take them out and move them around: think with them. But when, instead of moveable pieces, we have imprints on a tablet, we have to begin to imagine actions with commodities in ways unsupported by explicit motions of hand and eye. Presumably we then imagine those motions along with imagining commodities. Eventually there will be cuneiform notation for the names of the actions of grouping and dispersing objects. Where these sorts of actions are elaborated purely with placeholders in a dorsal-segregated way, there will come to be kinds of marks used to trigger imagined action in imagined token spaces: they will be symbols for arithmetical or logical operators. Then we have the notational basics for formal systems: we have a trained ability to evoke parietal function in isolation, we have an ability to imagine restricted or token or representational spaces by the use of those capabilities, we have representational objects that direct us to imagine groupings and positions of placeholders in those spaces, and we have representational objects that direct us to imagine actions that change those positions and groupings. Having the symbols in front of us, writing new ones, allows us to sustain chains of simulated action of indefinite length.
George Lakoff's cognitive philosophy of mathematics (Lakoff and Nunez 1997, Lakoff and Johnson 1999) is based on a related vision of mathematical practices as act metaphors derived from concerns with practical domains. Basic level schemas or grounding metaphors for arithmetic, Lakoff says, originate in "our intimate and precise understandings of domains like collecting, constructing objects, and moving" (1997, 34). Three basic schemas he identifies are: Arithmetic is Object Collection (subtraction, for instance, is taking smaller collections from larger collections to form other collections); Arithmetic is Object Construction (subtraction is taking smaller objects from larger objects to form other objects); and Arithmetic is Motion (subtraction of a given quantity is taking steps a given distance to the left, or backward). Basic practical schemas may be extended in ways that are mathematically useful but not part of the original practical base. An example is the extension of Sets are Objects in a Container to Sets are Objects (which may themselves be part of sets). Linking schemas are ways of reinterpreting mathematical practices based on one sort of practical activity into the mathematical terms based on another. Lakoff's example is this one: Corresponding to Arithmetic is Motion, with Numbers as Locations, there is a metaphor linking Arithmetic to Geometry, in which Numbers are Points on a line ... What is particularly interesting about this metaphor is that it is used to form a metaphoric blend, a composite of the source and target domains of the metaphor, that is, a composite of numbers and points on a line known as the number line (1997, 49). Lakoff goes on to describe the schema differences that underlie a number of puzzles and controversies in the philosophy of mathematics. A number series can be thought of as continuous, the way a gesture or eye motion is continuous; or it can be thought of as a row of points, like the rows of pebbles set out for numerical operations. Both conceptions will be entrenched as intuitions, since both are based on common human action. Mathematical novelty and reorganization can occur when mathematical practices thought in terms of one pragmatic base are systematically rethought in terms of mathematical practices understood in terms of another, as when algebraic practices were rethought in terms of a marked geometrical space. The view of mathematics that forms in this way is a view that integrates it with other sorts of simulational ability and other kinds of uses of representational artifacts. As with ASL, computer music, perspective drawings, and linguistic deixis, it is helpful to remember that the effective locus of representational action is the human body. One of the implications is that mathematical foundations are cognitive not formal, or, rather, that the epistemological foundations of mathematics should not be equated with logical foundations created post hoc (Irvine 1989). A further implication is that mathematical rigor is not in the formalism but in the trained, reliable ways a formalism is used.
In contrast with other representing practices, mathematical representation is solely and precisely concerned with matters of spatiotemporal relation and change, its linguistic forms, like the natural language locationals described by Talmy and by Landau and Jackendoff, evoking abstract simulational vision, somatosense and action without the sensory fullness of ventral participation. A link between mathematical and spatial aptitudes has often been demonstrated; strong correlation exists between mathematical talent and scores on spatial perception tests, "almost as if they were one and the same ability" (Dehaene 1997). Like the representational forms described earlier in this chapter, mathematics can also be seen as a representation-supported use of the parietal's basic spatiotemporal abilities. The psychology of mathematical localization has long had mathematical (and musical) genius pegged to "undue convolutional developments in the parietal lobule" (Critchley 1953, 56), and a right hemisphere IPL-frontal arc found active during sheerly spatial tasks like imagined rotations and paper folding, continues to be associated with giftedness in music, mathematics, physics and chess. Sandra Witelson is currently ascribing Einstein's preeminence in mathematical physics to an unusual formation of the inferior parietal areas. Einstein's postmortem cortex shows both hemispheres to be 15% wider than normal in the region of the IPL, with an unusual symmetry between hemispheres. Einstein's Sylvian fissure is also unusually positioned, connecting with the lower end of the central fissure so that the occipital-parietal-temporal junction behind the fissure is an unusually broad, integrated expanse. These two features suggest that, in Einstein's brain, extensive development of the posterior parietal lobes occurred early, in both longitudinal and breadth dimensions, thereby constraining the posterior expansion of the Sylvian fissure and the development of the parietal operculum, but resulting in a larger expanse of the inferior parietal lobe ... A consequence of this unusual morphology is that the full supramarginal gyrus lies behind the Sylvian fissure, undivided by a major sulcus, as is usually the case ... the compactness of Einstein's supramarginal gyrus within the IPL may reflect an extraordinarily large expanse of highly integrated cortex within a functional network. Witelson 1999, 2152 In the majority of people, who tend to have language lateralized to the left hemisphere, the Sylvian fissure is less steeply angled on the left, so there is more supramarginal tissue above the fissure than on the right. In Einstein's brain, which has the Sylvian fissure steeply angled on both sides, the left hemisphere is thus more like a right hemisphere in form. Since, as we have seen, the right inferior parietal is important to spatial presence and full-scale surrogate spatial imagining (while spatial function in the left tends to be language-related and token-scale, supporting minimal simulation), we can conjecture that Einstein's edge in physical thought may have been unusual concreteness and completeness of spatial imagining based in large regions of both hemispheres.
Dehaene suggests that subitization, the instantaneous, elementary count perception we share with babies and animals, depends on circuits of our visual system that are dedicated to localizing and tracking objects in space. The occipito-parietal areas of the brain contain neuronal ensembles that rapidly extract, in parallel across the visual field, the locations of surrounding objects. Neurons in these areas seem to encode the location of objects regardless of their identity and even to maintain a representation of objects that have been hidden behind a screen. Dehaene 1997, 68-9 We often say numbers are closer or more distant from each other. Dehaene believes use of a distance metaphor for quantity is intrinsic to our understanding of number, and that experimental results with basic arithmetic calculations are best understood if subjects are imagining numbers as points on a horizontal line. When experimental subjects are asked to compare pairs of small arabic digits and say which was larger, for instance, both response times and error rates vary with the degree of difference: As the numbers to be compared represented closer and closer quantities, the subjects' responses became increasingly slower and more error-prone. And when distance was kept constant, increasingly larger quantities again resulted in increasingly slower responses and larger error rates. Performance in comparing two numbers was determined by a Weber fraction, similar to that found when subjects have to compare physical parameters such as object size, line length, or tone height. Yet the number stimuli were presented in a symbolic notation, arabic numerals, whose surface form is largely arbitrary, conveying no information about number meaning. The distance effect suggested that subjects were converting the input numerals to an internal continuum, a mental " number line," and were performing a psychophysical comparison on this internal representation rather than on the surface form of the numbers. Dehaene 2000, 988 Dehaene further suggests that a particular area of the angular gyrus integrates arithmetic cognition based on the mathematical act metaphor he calls the number line (85-86). Lesions of the left IPL can result in Gerstmann's syndrome, a constellation of defects that includes an inability to calculate, along with a writing disability, difficulty distinguishing fingers of the right and left hands, and difficulty with the horizontal right-left axis in general. Functional imaging studies of arithmetic tasks find activity in areas damaged in Gerstmann's patients. On different runs, subjects were asked to name the digits, to compare them with 5, to multiply them by 3, or to subtract them from 11. In all subjects, fMRI identified a bilateral network that was very clearly pinpointed to the banks of the middle segment of the intraparietal sulcus, in a location in excellent agreement with the result of lesion studies in Gerstmann's syndrome. Dehaene 2000, 995 Gold (1995) suggests that all these characteristics of Gerstmann's syndrome may actually be defects in the perception of horizontal position. This understanding of Gerstmann's is in good general agreement with what we have seen about the role of the IPL in basic configural perception and action. It is in even better agreement with the particular role of the angular gyrus in perceiving or imagining groups of objects. Dehaene believes both hemispheres are able to set up a number line; in a patient with a callosal lesion, the normal distance effect was found for number comparisons performed by both hemispheres (2000, 990). Recent fMRI studies find bilateral parietal activation during calculation and comparison of pairs of numbers. Importantly, however, the size and lateralization of this parietal activation was modulated by task demands. Relative to letter reading, digit comparison yielded greater activity in the right inferior parietal area, multiplication greater activity in the left parietal area, and subtraction a bilateral increase ... digit naming in itself did not significantly activate the parietal areas ... Dehaene 2000, 995 Dehaene goes on to ask whether the number line function may be related to other spatial tasks: to attention shifts, hand movements, and imagined manipulation of objects. We do, after all, also talk about manipulating numbers; we carry the two. Does comparing quantities and calculating results require that we imagine we are looking back and forth between points on a right-left continuum? Carrying the two involves actual movement of the hand from the bottom to the top of a column, and doesn't it then seem as if the 2 we write had been transported to the place where we write it?
Conclusion: Constructing persons as knowledge |