Source
Automatically imported from: http://commons.somewhere.com:80/rre/1996/lessons.learned.in.tryin.html
Content
| | | | --- | --- | | Red Rock Eater Digest | Most Recent Article: Tue, 5 Dec 2000 |
lessons learned in trying to reform AI
``` I've enclosed a draft chapter I wrote for a book. Any comments would be appreciated. The first two paragraphs will probably get replaced, and the last section will get expanded. The audience is people interested in the intersection of AI and the social sciences (as opposed to the nearly null set of people who actually occupy that intersection) so the chapter will inevitably not be fully enjoyable to anybody. Let me know if anything seems unnecessarily obscure or offensive. It starts out by talking about military funding, a topic which is guaranteed to turn many people off because they've been exposed to reductionistic claims that military funding necessarily implies warlike intentions (as opposed to reductionistic claims that it's all science and we're not responsible if someone misuses it). The truth is in the middle, and we can't map the middle without powerful concepts.
Phil
Encl:
Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI
Philip E. Agre Department of Communication University of California, San Diego La Jolla, California 92093-0503 USA
phone: +1 (619) 534-6328 fax: +1 (619) 534-7315 net: pagre@ucsd.edu home: http://communication.ucsd.edu/pagre/
Draft chapter for Geof Bowker, Les Gasser, Leigh Star, and Bill Turner, eds, Bridging the Great Divide: Social Science, Technical Systems, and Cooperative Work, Erlbaum, 1996.
Version of 5 May 1996.
//1 Introduction
Every technology fits, in its own unique way, into a far-flung network of different sites of social practice. Some technologies, such as the cotton gin, are employed in a specific site, and in those cases we often feel that we can warrant clear cause-and-effect stories about the transformations that have accompanied them, either in that site or others. Other technologies, such as electric lighting and the telephone, are so ubiquitous -- found contributing to the evolution of the activities and relationships of so many distinct sites of practice -- that we have no idea how to begin reckoning their effects upon society, assuming that such a global notion of "effects" even makes sense.
Computers fall in this latter category of ubiquitous technologies. In fact, from an analytical standpoing, computers are worse than that. Computers are representational artifacts, and the people who design computers often start by trying to construct complete representations of the site where they will be used. This is the purpose of systems analysis, for example, and of the systematic mapping of conceptual entities and relationships in the early stages of database design. A computer, then, does not simply have an instrumental use in a given site of practice; the computer is frequently about that site in its very design. In this sense computing has been constituted as a kind of imperialism; it aims to reinvent virtually every other site of practice in its own image.
As a result, the institutional relationships between the computer world and the rest of the world can be tremendously complicated -- much more complicated than the relationships between the telephone world and telephone subscribers, or between the electric lighting industry and people who use electric lights in their workplaces and homes. The residents of these borderlands are many and varied, and increasingly so. They include the people who work on the border between the computer world and the medical world, whether because they conduct research in medical informatics or because they must encode their patient interactions for entry into an hospital's automated record keeping system. They likewise include the photographers whose livelihood is rapidly moving into digital media, the engineers who must employ computer-based tools for design rationale capture, and the social scientists who study the place of computers in society. Each of these borderlands is a complicated place; everyone who resides in them is, at different times, both an object and an agent of technical representation, both a novice and an expert. Above all, every resident of these borderlands is a translator between languages and worldviews: the formalisms of computing and the craft culture of the "application domain".
Every resident of the borderlands has a story, and in this chapter I would like to draw some lessons from my own. In 1988 I received a PhD in computer science at MIT, having conducted my dissertation research at the Artificial Intelligence Laboratory. I started out in school studying mathematics; I moved into computing because it allowed me to pay my school bills and because AI appealed to my adolescent sensibilities; I moved out of computing because I felt I had said everything I had to say through the medium of computer programs; and now I am some kind of critical social scientist concerned with the social and political aspects of networking and computing. This path has its geographical aspects, of course, and its institutional aspects; at each transition I was able to construct myself as a certain sort of person. Here, though, I wish to focus on the subjective aspects of my path. My ability to move intellectually from AI to the social sciences -- that is, to stop thinking the way that AI people think, and to start thinking the way that social scientists think -- had a remarkably large and diverse set of historical conditions. AI has never had much of a reflexive critical practice, any more than any other technical field. And so, even though I was convinced that the field was misguided and stuck, it took tremendous effort and good fortune to understand how and why. Along the way I spent several years attempting to reform the field by providing it with the critical methods it needed -- a critical technical practice.
This chapter describes what a critical technical practice in AI might be like. The first few sections will be concerned with AI as a field. Section 2 will recount some salient aspects of the field's institutional and intellectual history. Section 3 will sketch how the field understands itself; it is crucial to comprehend these self-understandings in order to work critically because they will shape the mainstream practitioners' perceptions of proposed alternatives. Section 4 will describe some of the discursive practices that have made AI so successful up to a point, while also making it difficult even to conceptualize alternatives.
The last few sections will describe my own experience and the lessons I have drawn from it. Section 5 will recount how I emerged from AI's unfortunately confining worldview and began to incorporate influences from philosophy, literary theory, and anthropology into my technical work. Section 6 will discuss what it means in practice to develop "alternatives" to an existing technical practice; for the most part, this notion is misleading. Section 7 will conclude with my own theory of critical engagement with a technical field.
//2 Historical constitution
The field of artificial intelligence arose in the years after World War II, when psychologists and others who had been pressed into wartime research returned to their labs. During the war, they had been greatly inspired by wartime technologies such as signal detection methods and tracking devices for guns. The attraction of these technologies was that they lent themselves to intentional description: a tracking device, for example, could be said to pursue goals and anticipate the future. As such, they provided the paradigm for a counterrevolution against behaviorism -- a way to make talk about mental processes scientific and precise.
Wartime research also created an important set of social networks among the military and civilian research communities. MIT in particular came to prominence as a technical university that had made significant innovations in the war effort, and after the war it found itself with extensive contacts in the suddenly much larger government research establishment. With the transition to the Cold War, this epistemic community united around a scientific and technical vision that Edwards (1995) has usefully identified as the "closed world" -- the whole world as one large technical system. Human beings, on this view, are themselves technical entities who serve as components of organizational systems: their bodies are machines and their minds are nodes in a hierarchical command-and-control network based on rational analysis and optimization.
As an institutional matter, then, AI was one part of an overall movement with a coherent worldview. It would be unfair to say that AI's founders were conforming themselves to military funding imperatives, just as it would be false to say that the emerging field of AI was intellectually autonomous. Numerous factors converged to reward the search for technologies and mathematical formalisms whose workings could be described using intentional vocabulary. Several technologies and formalisms received serious attention in this regard, including information theory, simulated neural networks, signal detection methods, cybernetic feedback systems, and computational complexity theory. As the field coalesced, though, only a handful of technologies and formalisms emerged to define the first decades of AI research. Most of these technologies and formalisms -- formal language theory, problem solving in search spaces, mathematical logic, and symbolic programming -- were qualitative in nature, not quantitative, and most of them were founded on the power and flexibility of the stored-program digital computer.
As the field of AI took form, it mixed elements of science and technology in complicated ways. Many of the founders of AI were psychologists, and they explained the field in terms of computer modeling of human thought processes. Others had a more abstract, less empirical interest in intelligence, and they explained the field with formulas such as, "building systems whose behavior would be considered intelligent if exhibited by a human being". Few of them regarded themselves as pure engineers seeking purely instrumental solutions to technical problems. Despite this, and notwithstanding the differences among individual investigators' approaches, the research programs at the new AI labs were deeply congruent with the broader military-scientific "closed world" consensus. As a result, the research in these labs continued for over twenty years with remarkably little detailed oversight or direction from ARPA and the other basic research funding agencies in the military. Broad research problematics such as problem-solving, learning, vision, and planning formed a kind of boundary between the individual researchers (and especially their students), who experienced themselves as having considerable autonomy, and their funding agencies, who had a ready vocabulary for explaining the relevance of this research to the agencies' overall technology strategy.
Critics of AI have often treated these well-funded early AI labs' as servants of the military and its goals. But it should be recognized that these labs' relative prosperity and autonomy within a deeply shared worldview also provided the conditions for dissent. In my own case, the first five years of my graduate education were paid by a fellowship from the Fannie and John Hertz Foundation. John Hertz, best-known as the founder of the Hertz car rental company, was a conservative patriot who left his fortune to a foundation whose purpose was to support military research. The Hertz Foundation fellowship program was, and is, administered largely by scientists at Lawrence Livermore National Laboratory who are associated with the military's nuclear weapons and ballistic missile defense programs. During the late 1970's, when I received my own Hertz Fellowship, the Hertz Foundation was aligned with the military's support for AI research. Numerous other graduate students in my laboratory were also Hertz Fellows, and the Foundation officers would speak explicitly about their hopes that the nation's research base -- whose health they understood in a broad sense, not simply in terms of immediate contributions to near-term military research programs -- would benefit from the large investment they were making in graduate students who were entering AI. The Hertz Foundation, and later ARPA, paid me a decent graduate-student salary during many months when I supplemented my technical work by reading a great deal of phenomenology and literary theory. The culture of MIT itself included a cult of "smartness", such that students deemed "smart" (including most everyone accepted to graduate school) were given very wide latitude to pursue their own research directions, no matter how odd they might seem to their worried dissertation committees. If the field of AI during those decades was a servant of the military then it enjoyed a wildly indulgent master.
Within academia, the early AI pioneers were largely engaged in a revolt against behaviorism. Behaviorism, in its day, had organized itself largely in reaction against the vague and unreproducible nature of introspectionist psychology. The metaphors provided by new technologies provided a means of placing mentalist psychology on a scientific basis, and a functionalist epistemology emerged to explain what it meant to offer mental mechanisms as explanations for experimental data. Although the participants in this intellectual movement experienced themselves as revolutionaries, mechanistic explanations of mind already had a long history, going back at least as far as Hobbes. Yet, in a curious twist, these mentalists acknowledged little inspiration from this tradition. Instead they reached back three hundred years to identify themselves with the philosophy of Descartes. Although Descartes' defense of a specifically mental realm was a useful symbol for mentalists engaged in polemics against behaviorism, the principal appeal of Descartes' theory was not its ontological dualism, but rather the explanatory freedom that Descartes' dualism afforded. The theorizing of later mechanists, from Hobbes to Locke to the associationists and reflex-arc theorists, was severely constrained by the limitations of their mechanical models. Descartes, on the other hand, could prescribe elegant and elaborate systems for the conduct of the mind without worrying how these systems might be realized as physical mechanisms. He intended his rules as normative prescriptions for rational thought; they were explanatory theories in the sense that they would accurately describe the thinking of anybody who was reasoning rationally. The stored-program digital computer, along with the theoretical basis provided by formal language theory and problem-solving search and the philosophical basis of functionalism, provided the pioneers of AI with a vocabulary through which normative accounts such as Descartes' could be rendered mechanical while also being meaningfully treated as mental phenomena, as opposed to physical ones.
//3 Self-conception
These aspects of AI's institutional and intellectual origins help to explain its very distinctive conception of itself as a field. The central practice of the field, and its central value, was technical formalization. Inasmuch as they regarded technical formalization as the most scientific and the most productive of all known intellectual methods, the field's prominent members thought of their research as the heir of virtually the whole of intellectual history. I have often heard AI people portray philosophy, for example, as a failed project, and describe the social sciences as intellectually sterile and empty. In each case their diagnosis is the same: lacking the extraordinarily precise and expressive methods of AI, these fields are inherently imprecise, woolly, and vague.
Any attempt at a critical engagement with AI should begin with an appreciation of the experiences that have made these extreme views seem so compelling. In its first fifteen years, AI developed a series of technical methods that provide interesting, technically precise accounts of a wide range of human phenomena. It is often objected that these machines are not "really" reasoning or planning or learning, but these objections can miss the point. The early demonstrations of AI were incomparably more sophisticated than the mechanistic philosophies of even a short time before. As a result, the people who had stayed up many late nights getting them to work felt justified in extrapolating this extraordinary rate of progress for one or two or three more decades at least. Critics of their research have often focused on particular substantive commitments that have seemed unreasonable, for example the frequent use of computer symbols such as REASON and DECIDE and GOAL whose relationship to the actual human phenomena that those words ordinarily name is suggestive at best. But here, again, it is important to recognize that AI's fundamental commitment (in practice, if not always in avowed theory) is not to a substantive position but to a method. Any particular set of mechanisms will surely prove inadequate in the long run, but it serves its purpose if it forces issues to the surface and sharpens the intuitions that will guide the development of the next mechanism along. AI people generally consider that their goals of mechanized intelligence are achievable for the simple reason that human beings are physically realized entities, no matter how complex or variable or sociable they might be, and AI's fundamental commitment (again, in practice, if not always in avowed theory) is simply to the study of physically realized entities, not to production systems or symbolic programming or stored-program computers.
In relation to the rest of the intellectual and technical world, then, AI long regarded itself as simultaneously central and marginal. It understood itself as central to human intellectual endeavor, and its integral connection to the closed-world agenda ensured that its main research centers (MIT, CMU, and Stanford) would number among the most prominent scientific laboratories in the world. But it was marginal in other, sometimes peculiar senses. Not only was it intellectually autonomous to a significant degree, but it was also a small world. Research results were communicated through internal report series, IJCAI (the biannual International Joint Conference on Artificial Intelligence), and the Internet. Its principal archival journal, Artificial Intelligence, was an important publication venue, but the central laboratories did not emphasize journal publication, and graduate students were often not taught how to write papers for journals.
The sense of marginality extended to the culture of the field. Much has already been written about the pecularities of the "hacker culture" (Turkle 1984, Weizenbaum 1976), with its extreme informality, emotional simplicity, resentment of externally imposed structures and constraints, and the leeway that the hackers afforded to one another's eccentricities. It was, paradoxically, an intensely social culture of seemingly quite asocial people. Whether explicitly or tacitly, they opposed the falseness of bureaucratic life to the principled meritocracy of their craft. Building things was truly the end purpose of the hacker's work, and everything about the methods and language and value system of the AI world was organized around the design and implementation of working systems. This is sometimes called the "work ethic": it has to work. The "result" of an AI research project is a working system whose methods seem original and broadly applicable; an "idea" is a method of building technical systems or a way of analyzing problems that motivates a promising system design; and a research "approach" is a conceptual and technical framework by which problems can be analyzed and transformed into a particular type of technical system. The field, accordingly, reckons its history primarily as a sequence of systems and secondarily as a history of debates among different approaches to the construction of systems.
It is commonly supposed that work in technical fields proceeds through sharply defined rational, logical reasoning. Many technical people actually believe this to be the case, but in AI at least, it is not true. The next section will describe some consequential fallacies in the field's ideas about precision and rigor, but it is equally important to understand the role of intuition in the AI's own explicit understandings itself as a technical practice. Whereas industrial computer programming is organized primarily around specifications that govern the input-output behavior of the various modules of a system, research programming in AI is self-consciously virtuosic and experimental. Much of the field's internal discourse has been concerned with the intuitions that guide the design of its complex, ambitious systems. The principle of modularity, for example, might be treated as an axiom or an instrumental expedient in industrial programming. But AI people understand modularity as a powerful but somewhat elusive principle of the universe, akin to a law of nature but much harder to define. The point is certainly not that AI people are mystics, or that they consciously wish to make anything obscure, but rather that they take seriously the craft nature of their work. AI people, likewise, are constantly discovering that different considerations trade off against one another. Modularity trades off against efficiency, for example, in the sense that systems can usually be made more efficient by breaking down the modularity boundaries that limit the amount of information that two components of a system can share. The expressive power of representation schemes trades off against efficiency as well, inasmuch as symbolic matching and inference tasks become rapidly less tractable as the representation languages provide a greater variety of ways of expressing equivalent concepts. Each of these broad generalizations can be made perfectly formal and precise in the context of particular, concrete design decisions, yet the generalizations themselves seem worthy of articulation and reification as lessons learned from research. The enormous obstinacy of technical work -- if a method cannot be made to work in a given case then no amount of sloppiness or vagueness will make it work -- seems to back these potentially nebulous intuitions with a "hardness" and irrefutability that philosophical or literary research never seems (to AI people anyway) capable of achieving.
//4 Discursive practices
The premise of AI, in rough terms, is the construction of computer systems that exhibit intelligence. One encounters different formulations of this premise at different labs, and from different individuals in the field. In philosophical and popular forums, the field is often discussed in terms of a seemingly fundamental question: can computers think? But little of the field's day-to-day work really depends on the answer to such questions. As a practical matter, the purpose of AI is to build computer systems whose operation can be narrated using intentional vocabulary. Innovations in the field frequently involve techniques that bring new vocabulary into the field: reasoning, planning, learning, choosing, strategizing, and so on. Whether the resulting systems really are exhibiting these qualities is hard to say, and AI people generally treat the question as an annoying irrelevance. What matters practically is not the vague matter of what the words "really mean" but the seemingly precise matter of how they can be defined in formal terms that permit suitably narratable systems to be designed. If you disapprove of the way that we formalize the concept of reasoning or planning or learning, they are likely to say, then you are welcome to invent another way to formalize it, and once you have gotten your own system working we will listen to you with rapt attention. If you disapprove of the very project of formalization, they are likely to say, then you are simply an obscurantist who prefers things to remain vague.
In an important sense, then, AI is a discursive practice. A word such as planning, having been made into a technical term of art, has two very different faces. When a running computer program is described as planning to go shopping, for example, the practitioner's sense of technical accomplishment depends in part upon the vernacular meaning of the word -- wholly arbitrary neologisms would not suffice. On the other hand, it is only possible to describe a program as "planning" is given a formal definition. The subfield of "planning research" consists of an open-ended set of technical proposals, joined by a densely organized family relationship but not by any a priori technical definition, about the implementable definitions that words in the semantic field around "planning" ("plan", "goal", "execution", "actions", "policies", and so forth) might be given. Different schools certainly differ in their standards of formalization, from "neat" (that is, explicitly and systematically mathematical) to "scruffy" (demonstrated simply through a compelling program). But they emphatically agree that the proof is in the programming, and that a proper research result consists in a method for casting planning-like tasks as technical problems that running computer systems can solve.
This dual character of AI terminology -- the vernacular and formal faces that each technical term presents -- has enormous consequences for the borderlands between AI and its application domains. The discourse of "domains" in AI is extraordinarily rich and complicated, and the field's practitioners take for granted a remarkable intellectual generativity. Once a term such as "planning" or "constraints" or "least commitment" has been introduced into the field through a first implemented demonstration in a particular domain, AI people will quite naturally shift that term into other domains, drawing deep analogies between otherwise disparate activities. Once an automated design problem, for example, has been analyzed into a large, discrete set of design choices, it immediately becomes possible to ask whether these choices can be made without backtracking -- that is, whether the choices can be made in some sequence in which earlier decisions never have unhappy implications for choices that must be made later on. Techniques that arose to support the patterns of backtracking that were discovered during research on story-telling may then find application in the automated design domain, or in a medical diagnosis domain, or in the domain of planning shopping trips. Having proven themselves broadly useful, these techniques might be abstracted into general-purpose algorithms whose computational properties can be studied mathematically, or they might be built into a programming language. Each technique is both a method for designing artifacts and a thematics for narrating its operation.
AI researchers can build computer models of reasoning in particular domains because their discourse is, in one sense, precise. But they can only make such a wide range of domains commensurable with one another because their discourse is, in another sense, vague. At any given time, AI's discursive repertoire consists of a set of technical schemata, each consisting of a restricted semantic field and a specific family of technical methods. Among the most prominent technical schemata are "planning" and "knowledge". Each of these words might be given a wide range of meanings in different cultural or disciplinary contexts. In AI, though, their meanings are closely tied to their associated technical methods, and they are not otherwise constrained. Absolutely any structure or purposivity in anybody's behavior, for example, can be interpreted as the result of planning. Miller, Galanter, and Pribram's Plans and the Structure of Behavior (1960), despite its lack of technical demonstrations, is nonetheless the field's original text of the rhetoric of planning. Absolutely any enduring competence, likewise, can be interpreted as the result of knowledge; John McCarthy's early papers (e.g., 1968 [1958]) provided one influential AI rhetoric of knowledge in terms of the predicate calculus.
The construction of an AI model begins with these most basic interpretations, and it proceeds systematically outward from them. Having detected an element of behavioral regularity in the life of some organism, for example, one can immediately begin enumerating the unitary elements of behavior and identifying those as the "primitive actions" that the putative planner has assembled to produce its plan. Miller, Galanter, and Pribram, motivated by Chomsky's linguistic formalisms and Newell and Simon's early problem-solving programs, helpfully suggested that all plans are hierarchical: a morning's activity might comprise several distinct activities (dressing, eating breakfast, answering correspondence), and each of those activities can be understood as themselves comprising distinct subactivities, which are themselves composite activities in turn, until finally one reaches a suitably elementary repertoire of actions from which all others are assembled. Miller, Galanter, and Pribram never offered a definitive set of these primitive actions, and the field has never felt it necessary to do so. (Schank's (1982) theory of the mental representation of action for purposes of story understanding, though, includes a fixed repertoire of primitive action types.) The purpose of the theory of planning has not been to provide a single technical specification for all domains, but rather to provide a set of technical schemata that can be expanded into a narrative thematics for any particular domain. Much of the practical work of AI, in other words, consists precisely in the deployment of these technical schemata to translate, or gloss, selected features of a given domain in terms that can also be interpreted as the operation of a computer program. The vagueness of AI vocabulary is instrumental in achieving this effect.
One consequence of this strategic vagueness, at least in my own experience, is that AI people find it remarkably difficult to conceptualize alternatives to their existing repertoire of technical schemata. The idea that human beings do not conduct their lives by means of planning, for example, is just short of unintelligible. At best it sounds like behaviorism, inasmuch as it seems to reject all possible theories of the mental processing through which anyone might decide what to do. The term "planning", in other words, exhibits an elastic quality: as a technical proposition it refers to a quite specific and circumscribed set of functionalities and algorithms, but as an empirical proposition it refers to anything at all that can plausibly be glossed with the term. This elasticity of meaning is already found in Miller, Galanter, and Pribram. Their formal definition of a "Plan" (they capitalize the term) is "any hierarchical process in the organism that can control the order in which a sequence of operations is to be performed" (1960: 16). A Plan is defined as a "process", and yet "process" is given no technical definition, either in their book or in subsequent planning research. Despite the broad and inclusive connotations of "any hierarchical process", in practice they use the word Plan much more specifically. In some places it refers to a "TOTE unit", which is a simple kind of feedback loop, and in other places it refers to a parse tree of the type described by formal language theory. This latter version has been the more influential, and virtually all AI planning theories interpret a "plan" as a symbolic datastructure that functions essentially as a computer program (another connotation that Miller, Galanter, and Pribram gesture at without formally embracing). As a result of this equivocation, attempts to deny the narrow technical theory sound to the ears of AI researchers like denials that the sequential ordering of human behavior is under the control of any coherent process at all.
Miller, Galanter, and Pribram's concept of a Plan also exemplifies another prominent feature of AI discourse: the tendency to conflate representations with the things that they represent. Their substantive theory is that behavior derives its structure from the structure of a Plan, and so they taught a generation of AI practitioners how to shift rapidly back and forth between talk about the structure of outward behavior and the structure of internal mental processes, and between the structure of these time-extended phenomena and the structure of static symbolic structures in the mind. This conflation of representations and worldly things is particularly encouraged by the domains that early AI research chose to illustrate its techniques. Newell and Simon's (1963, 1972) problem-solving research, for example, employed logical theorem-proving and puzzle-solving domains for which the distinction between mental representation and corporeal reality were shady. Proving logical theorems in one's head, after all, is a different activity from proving them with pencil and paper, but the essentially mathematical nature of the domain permits the distinction between logical propositions in working memory and logical propositions written on paper to be blurred. In particular, the mental representations readily capture everything about the real-world entities that can ever have any consequences for the outcome of the task. Of course, these domains appealed to early AI researchers in part because computer vision and robotics were very poorly developed, and these domains permitted research on cognition to begin without waiting on those other, possibly much more difficult research problems to be solved. But the privileged status of mathematical entities in the study of cognition was already central to Descartes' theory, and for much the same reason: a theory of cognition based on formal reason works best with objects of cognition whose attributes and relationships can be completely characterized in formal terms. Just as Descartes felt that he possessed clear and distinct knowledge of geometric shapes, Newell and Simon's programs suffered no epistemological gaps or crises in reasoning about the mathematical entities in their domains.
The conflation between representations and things can be found in numerous other aspects of AI research. It is found, for example, in the notion that knowledge consists in a model of the world, so that the world is effectively mirrored or copied inside each individual's mind. This concept of a "model", like that of a "plan", has no single technical specification. It is, rather, the signifier that identifies a technical schema: it provides a way of talking about a very wide range of phenomena in the world, and it is also associated with a family of technical proposals, each of which realizes the general theme of "modeling the world" through somewhat different formal means. Just as disagreements with the planning theory are unintelligible within AI discourse, it makes virtually no sense to deny or dispute the idea that knowledge consists in a world model. The word "model", like the word "plan", is so broad and vague that it can readily be stretched to fit whatever alternative proposal one might offer. AI people do not understand these words as vague when they are applied to empirical phenomena, though, since each of them does have several perfectly precise mathematical specifications when applied to the specification of computer programs.
//5 Waking up
My portrait of the AI community in the previous three sections is, of course, a retrospective understanding. Although they seem commonsensical to me now, and may seem commonsensical to others who have never been practitioners in the field, as an autobiographical matter I only came to these ideas through a long struggle. I had gone to college at an early age, having been constructed as a math prodigy by a psychologist in the region of the country where I grew up. (The arrival of court-ordered school integration in that region coincided with an emphasis on identifying talented students and grouping students into classrooms based on their test scores.) I began my college work as a math major before drifting over to the computer science department. My college did not require me to take many humanities courses, or learn to write in a professional register, and so I arrived in graduate school at MIT with little genuine knowledge beyond math and computers. This realization hit me with great force halfway through my first year of graduate school, and I took a year off to travel and read, trying in an indiscriminate way, and on my own resources, to become an educated person.
My lack of a liberal education, it turns out, was only half of my problem. Only much later did I understand the other half, which I attribute to the historical constitution of AI as a field. A graduate student is responsible for finding a thesis topic, and this means doing something new. Yet I spent much of my first year, and indeed the next couple of years after my time away, trying very hard in vain to do anything original. Every topic I investigated seemed driven by its own powerful internal logic into a small number of technical solutions, each of which had already been investigated in the literature. My attempts to investigate the area of concept learning, for example, endlessly converged back to a single idea: that all possible definitions of concepts form a mathematical lattice, and all reasonable inferences from evidence about a concept's correct scope could be analyzed in terms of lattice-theoretic operations of meeting and joining. This idea was already implicit in Winston's (1975) early research on concept induction, and had been fully worked through by others subsequently. It seemed inescapable, and overwhelmingly so.
With fifteen years' distance, I can now see that the idea of concept induction through lattice-crawling is indeed inescapable if one's ideas about concepts and evidence and learning are constrained by the ensemble of technical schemata that operated in the discourse and practice of AI at that time. But fifteen years ago, I had absolutely no critical tools with which to defamiliarize those ideas -- to see their contingency or imagine alternatives to them. Even worse, I was unable to turn to other, nontechnical fields for inspiration. As an AI practitioner already will immersed in the literature, I had incorporated the field's taste for technical formalization so thoroughly into my own cognitive style that I literally could not read the literatures of nontechnical fields at anything beyond a popular level. The problem was not exactly that I could not understand the vocabulary, but that I insisted on trying to read everything as a narration of the workings of a mechanism. By that time much philosophy and psychology had adopted intellectual styles similar to that of AI, and so it was possible to read much that was congenial -- except that it reproduced the same technical schemata as the AI literature. I believe that this problem was not simply my own -- that it is characteristic of AI in general (and, no doubt, other technical fields as well). This is not to say that AI has no intellectual resources and no capacity for originality. In recent years particularly, the field has made productive connections with a wide variety of other technical fields, establishing common cause through the sharing of technical schemata.
My own route was different. I cannot reproduce its whole tortuous detail here, and so it will inevitably sound simpler in the retelling than it was in the living. Nor will I analyze my own personal emotional investments in the process. But the clarity of hindsight makes evident that I drew on the internal resources of the field, even as I struggled to find my way out of it. I began by filling my notebook with exhaustively detailed stories from my own everyday life. By this time I had grown preoccupied with planning research, which struck me as wildly counterintuitive even though I could not explain how or why, and so I decided to gather some examples of real-life planning. In doing so, I was following an AI tradition of introspection that has been described aptly, if unsympathetically, by Turkle (1984). Many of the early AI researchers were clearly attempting, at one level or another, to reproduce their own psyches on computers, and many of them drew on introspection to motivate their programs. Introspection as a formal research method in psychology, of course, had been comprehensively discredited decades earlier. But AI people have not regarded introspection as evidence but as inspiration; since the functionality of their computer systems provides a fully adequate criterion of the success of their research, it does not matter what experiences might have motivated the systems' design. And introspection is close at hand.
But my own practice was different from introspection in one important respect: whereas introspection attempts to observe and recount mental processes under specially controlled conditions, I was trying to remember and recount episodes of concrete activity that took place in my own everyday life. Together with my fellow student David Chapman, I rapidly developed a method that I called "intermediation". Having noticed some interesting sequence of events in the course of washing the dishes or carrying out the trash, I would write it down in my notebook in as much detail as I could remember. Along the way, I would invent names for aspects of the recounted activity that seemed relevant to some technical concern. The method worked best if these names were intermediate in their degree of abstraction, thus the term "intermediation". For example, I became interested in what I called "hassles", which are small bits of trouble that recur frequently in routine patterns of activity. Having noticed a hassle, for example an episode in which silverware tried to crawl into the garbage disposal while washing dishes, I would write out in some detail both the episode itself and the larger pattern's attributes as a hassle. Having done so, I found that I would then start spontaneously noticing hassles in other activities, particularly hassles that were analogous in some way to the hassles that I had already noticed and written out in my notebook. I did this regularly for a couple of years, to such an extent that I was continually noticing various aspects of the mundane mechanics of my daily life; I was also continually noticing the many small transformations that my daily life underwent as a result of noticing these things. It is hard to convey the powerful effect that this experience had upon me; my dissertation (Agre 1988), once I finally wrote it, was motivated largely by a passion to explain to my fellow AI people how our AI concepts had cut us off from an authentic experience of our own lives. I still believe this.
Perhaps someday I will finally write out my treatise on the true functioning of everyday routine activities, based on the great mass of anecdotes that I accumulated by this procedure. My purpose here, though, is to describe how this experience led me into full-blown dissidence within the field. Given that an AI dissertation is based on a computer program, my investigations of everyday routine activities were always aimed at that goal. I wanted to find an alternative means of conceptualizing human activity -- one that did not suffer the absurdities of planning but that could be translated into a working demonstration program. To this end, I spent many months working back and forth between concepts to describe everyday activities and intuitions that seemed capable of guiding technical work. Most of these intuitions would be impossible to explain without developing an elaborate apparatus of concepts, and indeed I found that my thinking about these matters had become impossible to explain to anybody else. A small number of my friends, most notably David Chapman, sat still for long, complex explanations of the phenomena I was observing and the intuitions that they seemed to motivate. But clearly I had to bring this project back down to earth.
In order to find words for my newfound intuitions, I began studying several nontechnical fields. Most importantly, I sought out those people who claimed to be able to explain what is wrong with AI, including Hubert Dreyfus and Lucy Suchman. They, in turn, got me started reading Heidegger's Being and Time (1961 [1927]) and Garfinkel's Studies in Ethnomethodology (1968). At first I found these texts impenetrable, not only because of their irreducible difficulty but also because, as I mentioned earlier, I was still tacitly attempting to read everything as a specification for a technical mechanism. That was the only protocol of reading that I knew, and it was hard even to conceptualize the possibility of alternatives. My first intellectual breakthrough came when, for reasons I do not recall, it finally occurred to me to stop translating these strange disciplinary languages into technical schemata, and instead simply to learn them on their own terms. This was very difficult because my technical training had instilled in me two polar-opposite orientations to language -- as precisely formalized and as impossibly vague -- and a single clear mission for all discursive work -- transforming vagueness into precision through formalization (Agre 1992). The correct orientation to the language of these texts, as descriptions of the lived experience of ordinary everyday life, or in other words an account of what ordinary activity is like, is unfortunately alien to AI or any other technical field.
I still remember the vertigo I felt during this period; I was speaking these strange disciplinary language, in a wobbly fashion at first, without knowing what they meant -- without knowing what sort of meaning they had. Formal reason has an unforgiving binary quality -- one gap in the logic and the whole thing collapses -- but this phenomenological language was more a matter of degree; I understood intellectually that the language was "precise" in a wholly different sense from the precision of technical language, but for a long time I could not convincingly experience this precision for myself, or identify it when I saw it. Still, in retrospect this was the period during which I began to "wake up", breaking out of a technical cognitive style that I now regard as extremely constricting. I believe that a technical field such as AI can contribute a great deal to our understanding of human existence, but only once it develops a much more flexible and reflexive relationship to its own language, and to the experience of research and life that this language organizes.
//6 The fallacy of alternatives
These foreign disciplinary languages were beginning to provide an established vocabulary for the intuitions that I had developed by noticing and writing out episodes of routine activity. In broad outline, my central intuition was that AI's whole mentalist foundation is mistaken, and that the organizing metaphors of the field should begin with routine interaction with a familiar world, not problem-solving inside one's mind. In taking this approach, everything starts to change, including all of the field's most basic ideas about representation, action, perception, and learning. When I tried to explain these intuitions to other AI people, though, I quickly discovered that it is useless to speak nontechnical languages to people who are trying to translate these languages into specifications for technical mechanisms. This problem puzzled me for years, and I surely caused much bad will as I tried to force Heidegger-speak down the throats of people who did not want to hear it. Their attitude was: if your alternative is so good then you will use it to write programs that solve problems better than anybody else's, and then everybody will believe you. I knew that this stance was wrong, but I did not understand how.
I now believe that it is wrong for several reasons. One reason, which I have already discussed above, is that AI discourse makes it exceptionally difficult to conceptualize alternatives to the field's prevailing ideas. Write a program whose operation you understand in different terms, and somebody will observe that your program can perfectly well be described as having knowledge, executing plans, and so on. Never mind, then, that you choose to talk about the program differently; in fact, they will say, it is nothing new.
Even more profoundly, I think that the very concept of "alternatives" is misleading if not wrong-headed. AI's existing language and technical practice, like any disciplinary culture, runs deeper than we are aware. Having been socialized into the field, I accumulated an extensive network of linguistic forms, habits of thought, established techniques, ritualized work practices, ways of framing questions and answers, genre conventions, and so forth. It is impossible to simply cast off that whole network of cultural forms, any more than I could simply decide to stop being American and start being Thai, or to become transcendentally stateless and cultureless. As a result, attempts to formulate an alternative worldview for AI, or even to secede from the field altogether, are bound to fail.
I began to understand this once I had attained a few years' critical distance on the two computer programs that I described in my dissertation (Agre 1988) and more recently in my book (Agre in press). I wrote the first of these, a program called RA that operates in a conventional AI "blocks world", as an experiment in computational improvisation; rather than constructing a plan which it then executes wholesale, it conducts a fresh reasoning-through of its situation as quickly as it can. David Chapman wrote the second, with some participation from me in the later stages of development, a program called Pengi that played a video game by a similar improvisation, but with a much more sophisticated model of visual perception.
Although intended as alternatives to the conventional theories of planning, reasoning, and vision, these programs ultimately turned out to recapitulate some subtle confusions of the conventional methods. Specifically, both of these programs relate to their "worlds" from a bird's-eye view -- or, more precisely, from an orthographic projection from the side, as if the simulated agent were infinitely far away from the action except for the one instrument through which it moves things around: a cartoon "hand" for RA and a cartoon penguin for Pengi. Orthographic projections are ubiquitous in the diagrams of AI papers; they make it seem reasonable that the simulated agent maintains a panoptic representation of its entire environment. This is one particularly insidious manifestation of AI's tendency to conflate representations and things-represented. In the case of RA, this conflation was already hidden by a convention of blocks world: the blocks have their names ("A", "B", "C", etc) written on them, as if the agent's mental symbols were part of the material world, automatically connected to the things they name. In the case of Pengi, the problem was much more subtle. Because Pengi employs a somewhat realistic model of vision, it did not automatically represent the whole of reality to itself. But the adversarial and partially random nature of the video game meant that Pengi could not rely on stable structures in the world or large-scale patterns in its interactions with the world. And since Pengi had no body, and thus no strong sense of being located anywhere in particular, every part of the visible "world" was potentially relevant all of the time. As a result, Pengi (like any player of a video game) was forever scrambling to allocate its attentional resources to the most important objects in its visual world. It could carry on reasonably well by focusing primarily on the objects closest to it. It could only keep track of individual objects, however, by physically tracking them across the screen, much as RA kept track of blocks by knowing their names. In the end, therefore, Pengi did not provide the clear-cut alternative that we had hoped.
Neither of these programs was a failure. To the contrary, each of them introduced technical methods that may have some lasting value. And each of them does point to the utility of different metaphors for the technical work of the field. Nonetheless, each of them reflects the inherent difficulty of inventing an "alternative" to established technical methods. It is useful, by way of summary, to distinguish four kinds of difficulty. First, it is difficult to become aware of the full range of assumptions underneath existing practices, from technical methods to genre conventions to metaphors. Second, having formulated an alternative intuition, it is difficult to reduce that intuition to a novel technical method -- a new type of artifact, or a new way of building artifacts. Third, having invented a new technical method, it is difficult to prevent it from being understood within the elastic boundaries of existing technical schemata. Fourth, having coupled a new technical method with a new way of talking about the phenomena, it is difficult to apply the method to any real cases without inventing a lot of additional methods, since any worthwhile system will require the application of several interlocking methods, and use of the existing methods may distort the novel method back toward the traditional metaphors.
//7 Critical engagement
When I first started trying to reform AI, I believed in revolutions. It seemed to me that I could clear the ground completely and start over, working out a whole alternative intellectual system that would replace everything that was there before. The concept of a generative metaphor seemed to hold out considerable promise in this direction, given that so much of the underlying substantive problem with AI really can be understood as expressing a single principle, mentalism as opposed to interactionism.
Now, though, I do not believe that it works that way. Instead, I believe in hermeneutics. The purpose of technical exercises, aside from the utility that they might actually have in application to real problems in the world, lies precisely in their limitations and failures. Perhaps we can learn to approach technical work in the spirit of reductio ad absurdum: faced with a technical difficulty, perhaps we can learn to diagnose it as deeply as possible. Some difficulties, of course, will be superficial and transient. But others can serve as symptoms of deep and systematic confusions in the field. We are only aware of this possibility if we develop the critical tools to understand the depths below the ordinary practices of a technical field. Some of these critical tools will draw on the methods of institutional and intellectual analysis that have served generations of philosophers, sociologists, and literary critics. Others may be sui generis responses to the specific properties of technical work. Research could proceed in a cycle, with each impasse leading to critical insight, reformulation of underlying ideas and methods, fresh starts, and more instructive impasses.
But the language of hermeneutics is not adequate, either, because it suggests a solitary "reader" facing the practical reality of technical work as an individual. Technical work is performed in and by communities, and a critically engaged practitioner cannot hope to found an alternative community in which everyone shares the same critical premises and methodologies. As I worked my way toward a critical technical practice, this was the part that I found hardest: maintaining constructive engagement with researchers whose substantive commitments I found wildly mistaken. It is tempting to start explaining the problem in an alien disciplinary voice, quoting phenomenology or dialectics as an exogenous authority, but it is essentially destructive.
The constructive path is much harder to follow, but more intellectually rewarding. Its essence is to evaluate a research project not by its correspondence to one's own substantive beliefs but by the rigor and insight with which it struggles against the patterns of difficulty that are inherent in its design. Faced with a technical proposal whose substantive claims about human nature seem mistaken, figure out what deleterious consequences those mistakes should have in practice. Then find out if the predicted impasses have actually arisen. Perhaps they have, and research has proceeded on the basis of an insufficiently radical interpretation of their significance, leading to new and different problems. Or perhaps they have not, and lessons can be learned from the precise type of confusion that their research has been exhibiting as it has managed to overlook the impasses inherent in its approach. These types of confusion, as I have pointed out in discussing both my own work and others', are many and varied. Confusions can be buried in vague or ambiguous language, in notational conventions, in experimental designs, and in many other places as well. Critical methods from other fields might be helpful in discovering the many types of confusion that have presumably gone unrecognized thus far. But nothing can substitute for the daily work of trying to get things built and working. This is probably not my own path any more, but perhaps it will be a path for others in the future.
A critical technical practice will, at least for the foreseeable future, require a split identity -- one foot planted in the craft work of design and the other foot planted in the reflexive work of critique. Successfully spanning these borderlands, bridging the disparate sites of practice that computer work brings uncomfortably together, will require a historical understanding of the institutions and methods of the field, and it will draw on this understanding as a resource in choosing problems, evaluating solutions, diagnosing difficulties, and motivating alternative proposals. More concretely, it will require a praxis of daily work: forms of language, career strategies, and social networks that support the exploration of alternative work practices that will inevitably seem strange to insiders and outsiders alike. This strangeness will not always be comfortable, but it will be productive nonetheless, both in the esoteric terms of the technical field itself and in the exoteric terms by which we ultimately evaluate a technical field's contribution to society.
//* References
Philip E. Agre, The dynamic structure of everyday life, PhD dissertation, Department of Electrical Engineering and Computer Science, MIT, 1988.
Philip E. Agre, Formalization as a social project, Quarterly Newsletter of the Laboratory of Comparative Human Cognition 14(1), 1992, 25-27.
Philip E. Agre, Computation and Human Experience, Cambridge: Cambridge University Press, in press.
Paul N. Edwards, The Closed World: Computers and the Politics of Discourse in Cold War America, Cambridge: MIT Press, 1996.
Harold Garfinkel, Studies in Ethnomethodology, Polity Press, 1984. Originally published in 1967.
Martin Heidegger, Being and Time, translated by John Macquarrie and Edward Robinson, New York: Harper and Row, 1961. Originally published in German in 1927.
John McCarthy, The advice taker, in Marvin Minsky, ed, Semantic Information Processing, Cambridge: MIT Press, 1968. Originally published in 1958.
George A. Miller, Eugene Galanter, and Karl H. Pribram, Plans and the Structure of Behavior, New York: Holt, 1960.
Allen Newell and Herbert A. Simon, GPS: A program that simulates human thought, in Edward A. Feigenbaum and Julian Feldman, eds, Computers and Thought, New York: McGraw-Hill, 1963.
Allen Newell and Herbert Simon, Human Problem Solving, Englewood Cliffs, NJ: Prentice-Hall, 1972.
Roger C. Schank, Dynamic Memory: A Theory of Reminding and Learning in Computers and People, Cambridge University Press, 1982.
Sherry Turkle, The Second Self: Computers and the Human Spirit, New York: Simon and Schuster, 1984.
Joseph Weizenbaum, Computer Power and Human Reason: From Judgement to Calculation, San Francisco: Freeman, 1976.
Patrick H. Winston, The Psychology of Computer Vision, New York: McGraw-Hill, 1975. ```
| | | --- | | ProcessTree Network TM For-pay Internet distributed processing. | | Advertising helps support hosting Red Rock Eater Digest @ The Commons. Advertisers are not associated with the list owner. If you have any comments about the advertising, please direct them to the Webmaster @ The Commons. |