‘Beginning to reason is like stepping onto an escalator that leads upward and out of sight. Once we take the first step, the distance to be travelled is independent of our will and we cannot know in advance where we shall end.’

— Peter Singer (1982)

‘You seriously believe a universe in which billions of sentient beings on this planet alone die horrible deaths in war, famine and plauge [sic], a universe in which people have a fleetingly short time to live before their health and strength drain away, a universe in which misunderstanding is endemic and barriers between minds are unbreechable, a universe in which most sentients are forced to serve others to obtain the basic necessities of life and in which a great many live under fear and repression, a universe where the only known planet with sentient life is under constant threat of being wiped clean by a whole range of disasters, where the entire population is under some degree of harmful delusion, doesn’t suck?’

— Michael Wilson

There are many situations where we lack knowledge to move the world from an undesired state to a more desired state. We call these situations problems. In the following, I describe what I see as one of the major challenges that a technical solution to the problem of problem solving faces.

If we cannot ignore a problem, what we do is this: We take our limited knowledge about the world, our limited knowledge about what we want, and use our limited reasoning capabilities to find out what we should do to move the world closer to how we want it to be. Given that we believe the world to be like this, and given that we would like the world to be like that, we infer what action we should take.

In general, inference denotes the process of assuming that certain statements about the world are true and deriving what follows for the truth of other statements. Machines are potentially much better at inference than the human mind is. Programs have the potential to encode much more and much more precise knowledge than any one of us could learn in a lifetime. The laws of probability theory and approximations thereof can be used to infer precise knowledge about the world from data and to reason using this knowledge.

The fact that we can use machine inference to solve problems that are too difficult to be solved using human reasoning makes research into inference methods important. When we use machine inference to figure out what the dynamics of protein folding are in order to solve diseases like Alzheimer’s, we do so because, given all our data and our wish to cure the disease, our human minds still cannot figure out the cure on their own. A prerequisite for the use of machine inference is to have the problem statement available formally, as a program, a mathematical object. By singling out a small, well-defined problem, we can formally write down knowledge about the problem domain (or a program for inferring such knowledge from data) and a program that uses this knowledge to solve a particular reasoning task, i.e. to help us in determining how to change things for the better.

Any such small, formal problem statement captures only very little about what we want things to be like more generally. What do we want things to be like? We know that there are some things we want because they lead to other good things — these we call instrumental values — and there are some we want for their own sake — terminal values. We can make guesses about what our terminal values are, what we value for its own sake — joy, freedom, discovery, beauty, kindness — but ultimately, what we want the world to be like is not summed up well by any (or all) of the individual concepts behind our guesses. The name for that which does capture all about what we want is preference.

Because preference is stored opaquely in our brains, we cannot directly access its content, we can only use it to some extent. Similarly, we have little access to how our minds represent concepts like “tree”, “word”, or “spring”. This does not mean that there is no precise structure behind any given concept. On the contrary, the theories that best predict human concept learning and reasoning in recent psychological experiments are those that assume that concepts are represented as probabilistic programs.

Likewise in the case of preference: Our meager introspective abilities obscure the fact that the term ‘preference’ denotes a precise informational structure. This is easy to see where limitations of reasoning make us more uncertain than dictated by what can be deduced from the information that we know is stored in our brains, but is likely to be true more generally.

Take our search for a cure for Alzheimer’s: We have an intuitive idea what results are good and what results are bad — those that actually cure the disease and that do so without side-effects are the good ones. Nonetheless, we are uncertain about how our preferences rank different states of the world; without using machine inference to figure out the dynamics of protein folding, we do not know how preference orders possible states of the world because we cannot tell which state corresponds to a cure and which to a useless substance. By using machine inference to determine which state corresponds to a cure and which does not, we factor out a small part of our preferences that we assume to be independent from the rest (although it is not!). We thus hope to improve our understanding of what this part of our preferences says about the world.

When we — or our machines — work towards the solution of any particular subproblem without taking into account our preferences as a whole, we commit what could be called a mistaken factorization of preference. Preference as a whole makes a statement about what is the best choice at any given point in time, and if we look at only a small part of this statement, we lose value. On the other hand, if we can access the formal statement that our preferences make as a whole, then we may be able to use the reasoning capabilities of machines to determine much more precisely what the best choice looks like than would be possible through introspection.

If preference is a mathematical structure — even if it is currently implemented in our brains in a distributed and implicit way — then what kind of structure could it be? I do not know the answer to this question, but there are situations that are similar in the sense that they also take an intuitive idea and reify it into a mathematical object.

In computer science, there is the notion of the future of a computation. For example, at point # in the program (* 3 (+ 4 #)), what the future holds is that whatever value we hand it, it will add 4, multiply the result by 3, and then do whatever it does to the return value of a program that has finished, e.g. print the result to the screen. The notion of a continuation captures the idea of taking the future of a computation and storing it in an object. If we capture the continuation at #, the future starting from this point becomes a mathematical object, a value that we can pass around just like any other object and that we can reason about formally.

Analogously, we would like to take the diffuse notion of the preferences of a decision-making system (like you and me) and reify it into a formal object. And analogously, we expect this object to be a computational structure that contains information about what will — or, in the case of preference, should — be done in the future, but it may take lots of computation to determine what exactly this information says.

The project of formalizing preference has two parts: understanding the structure of preference (i.e. what kind of object are preferences, how do they compose) and getting at the actual content of human preference (i.e. extracting or pointing to the preferences of a given agent).

There are proposals for what the structure of preference could look like (e.g. preference logics, utility theory), but they seem insufficient in non-trivial situations. Two examples of such situations are (1) that we want our preferences not to lose meaning when it turns out that we have been mistaken about some of the things our preferences talk about (the so-called ontology problem), and (2) that we may have preferences about how we want preferences to interact. Different people appear to want different things, and even within a single mind, seemingly contradicting wishes exist. For example, there are things we want to do and there are things we actually enjoy doing, and these are often not the same. How do we figure out the statement that such a system of preferences makes about what should be done? This is called preference aggregation across agents and could be called compositionality of preference within a single agent.

Formalizing the content of our preferences, i.e. pointing to preference in a precise, machine-readable way, poses similarly challenging problems. The strongest illustration of this that I can currently think of is the following: If our preferences determine which method of formalizing their content is the correct one (namely the one that results in our actual preferences), and if we cannot know or use our preferences with precision until they are available as mathematical objects, then how can we find the correct formalization method? I can imagine that knowing the structure of preference would clarify what properties a method needs to have that allows us to formally refer to the preference content of any given agent, but to what extent this is the case is an open question.

To summarize, the fundamental problem is this: We have only limited access to what we want, and we cannot really figure out what follows from that which we do know about what we want. Machines are potentially much better at reasoning about what follows if we can give a formal description of what we want. However, if we formalize only a few small problems, we lose value due to our limited reasoning about the remaining part of our preferences and due to assuming independence between preferences when in reality they are intertwined. We need to understand preference as a formal object if we want to use machine inference to figure out what should be done to make this world a nicer place.

I thank Vladimir Nesov for useful discussion and for originating many of the ideas mentioned here.

Irgendwo hier liegen die Grenzen des Mustererkennungsapparats in unserem Kopf. Das, was wir verstehen können, ist keine obere Schranke für die Komplexität unserer Welt. Aber ich wiederhole mich.

Here are the slides for the presentation I held in Tuesday’s philosophy class, in the hope that they may be of some use, even if part of it is incomprehensible for anyone who did not read the paper or listen to the talk:

You probably heard of the idea that, at some point in time, we might create systems that solve certain tasks and that get better at these tasks by recursively modifying their code. Here is some scary reasoning:

1. A system cannot predict (=understand) a system of greater algorithmic complexity.
2. Therefore, the only way for a system to improve in a way that increases its algorithmic complexity is trial and error, thereby keeping the best results — i.e. evolution.
3. The only goal that is stable under evolution is rapid reproduction.
4. Therefore, the only stable goal for recursively self-improving systems is rapid reproduction.

I really hope that someone will point out the flaw in this line of thought or show me the reason why it does not apply to our world and to any self-modifying systems we might create.

Wir Menschen unterscheiden uns nicht großartig in unseren Wünschen. Wir wollen Glück, Freude, Freiheit, Unabhängigkeit, Sicherheit, Wissen, Kreativität, Individualität, Sexualität, Freundschaft und Liebe (nun ja, Männer zumindest). Wir schätzen unser Leben, das unserer Freunde, unserer Familie und das unserer sechs Milliarden Mitmenschen. Trotzdem ziehen wir in verschiedene Richtungen, konkurrieren, intrigieren und machen generell den Eindruck, als ob wir es darauf anlegen, paradox zu handeln.

Wenn wir verstehen, welches Ausmaß die Zukunft hat, die auf dem Spiel steht, und wenn wir uns im Großen und Ganzen einig sind, was uns jetzt und für diese Zukunft wichtig ist, warum funktioniert es dann nicht besser^TM?

Warum leben wir nicht längst in Utopia, wenigstens asymptotisch?

Die Erklärung, die ich nicht glaube: Es geht nicht besser. Würde man jeden Menschen fragen, wie sehr diese Welt seinen Vorstellungen entspricht, und so zu einem Gesamtbild kommen, so gäbe es nichts, was dieses Bild dauerhaft besser machen könnte. Für diese Erklärung spricht die Anpassungsfähigkeit unseres Gehirns, die daran schuld ist, dass die meisten Änderungen unsere Gesamtzufriedenheit nicht dauerhaft verbessern. Glück ist die erste Ableitung positiver Veränderung. Aber, erstens: Lasst uns die offensichtlichen Unmenschlichkeiten dieser Welt beheben, dann können wir noch einmal darüber reden, ob es nicht besser geht. Zweitens: Manche Leute scheinen immer ein bisschen glücklicher zu sein als andere. Gene und Umwelteinflüsse legen die Biochemie unseres Gehirns fest und wir sind dabei, beides zu verstehen.

Die Erklärung, die ich gerne glauben würde: Die Probleme unserer Welt sind komplex. Wir sind auf dem Weg zu Lösungen, aber die erfordern ein gewisses Mindestmaß an Zeit und Technologie. Es wäre falsch, sich an neue Technologien zu klammern, weil diese beinahe immer zu polaren Zwecken eingesetzt werden können, aber ein Blick auf die Geschichte macht klar, dass neue Technologien Einfluss haben. Die Kombination aus omnipräsentem mobilem Web für die Massen und Suchmaschinen, die natürliche Sprache verstehen, könnte die Wissensverteilung weiter demokratisieren. Prognosemärkte (die von Google, Microsoft, HP und Intel bereits intern eingesetzt werden) könnten Teile der Politik rationaler gestalten, der Anfang der vollständigen Aufzeichnung der Menschheitsgeschichte alle kollektiven Entscheidungen.

Die Erklärung, die immer nur andere betrifft: Das sind alles egoistische Nichtsnutze, denen die Menschheit egal ist, so lange sie Familie, Job und ein halbwegs interessantes Leben haben. Unterstützt werden sie in ihrer Haltung von Wissenschaft und Wirtschaft, die Gedanken über den Lauf der Welt zugunsten kurzfristiger und handfester Resultate bestrafen. Andererseits werden gesellschaftliche Fragen gerne mal eben beim Mittagessen gelöst (wenn gerade keine Fußball-WM stattfindet) und mit zufriedenem “Tja, so müsste man’s machen” abgehakt. Zu Handlungen kommt es natürlich nicht, denn dafür bräuchte man Lösungen, die tatsächlich funktionieren, müsste herausfinden, wie man als einzelner zur Umsetzung beitragen kann, und müsste die Lösungen finden, von denen man selbst profitiert. Wozu die Menschheit retten, wenn es nicht entweder Geld, Sex oder Status bringt oder sowieso auf dem Weg zur Rettung des eigenen Lebens liegt?

Die Erklärung, die mich (und dich!) betrifft: Wir arbeiten auf Teilziele hin, die nicht direkt dem entsprechen, was wir wirklich wollen. Weil das fast jeder tut, weil verschiedene Teilziele oft gegensätzliche Aktionen erfordern und weil die Ziele selbst dann oft nicht erreicht werden, heben sich unsere Bemühungen mehr oder weniger auf. Unser Tun führt so zwar zu neuen Methoden und zu neuen Erkenntnissen über unsere Welt, die indirekt zur Realisierung unserer Wünsche beitragen können, ist aber ineffektiv und potentiell schädlich. In dem Moment, in dem wir uns einer Ideologie verschreiben, weil wir glauben, dass die Durchsetzung von deren Axiomen den Menschen das geben wird, was sie wirklich wollen, arbeiten wir an der Verbreitung der Ideologie und nicht mehr an den eigentlichen Problemen.

Glücklicherweise ist die Lösung einfach: Wir wählen in jedem Moment die Handlung, die für sich genommen am ehesten unseren Werten entspricht, anstatt uns auf eine Ideologie oder auf ein langfristiges Ziel festzulegen und darauf hinzuarbeiten.

Dummerweise funktioniert sie nicht in jedem Fall, insbesondere dann nicht, wenn wir existentielle Risiken — Katastrophen, die das Ende der Menschheit bedeuten können — in Betracht ziehen und uns der Fortbestand der Menschheit doch ein bisschen kümmert.

KI in zwei Sätzen: Die Annahme, dass wir in absehbarer Zeit auf einen relativ allgemeinen Mustererkennungsalgorithmus stoßen, der mit genügend Rechenpower die Mustererkennungs- und Vorhersagefähigkeiten des menschlichen Gehirns übertrifft, ist (für diese Art von Annahmen) weit verbreitet. Deutlich kontroverser ist die Idee, dass Algorithmen praktisch möglich sein könnten, die Veränderungen an sich selbst vornehmen, um so große Klassen von formalisierbaren Probleme bestmöglich zu lösen — unabhängig davon, wie anspruchsvoll diese Probleme sind, d.h. wie viel Intelligenz zu deren Lösung nötig ist.

Die formale Analyse der Approximierbarkeit theoretischer Modelle von Superintelligenz in unserer physikalischen Welt benötigt unsere Aufmerksamkeit, wenn wir wissen wollen, wo auf unserer Liste existentieller Risiken und Chancen maschinelles Lernen steht. Forschung auf dem Gebiet ist ein langfristiges Vorhaben, eines, das jahrelanges Lernen voraussetzt und das mit signifikanter Wahrscheinlichkeit fehlschlägt. Das ändert nichts daran, dass solche Forschung wirklich, wirklich wichtig ist.

Letzte Woche, bei Pasta und Pizza, hat Jürgen die Frage in die Runde geworfen, wie groß denn der Anteil unserer Zeit sei, den wir für das Jetzt leben, und wie groß der, den wir für die Zukunft leben. Zunächst allgemeine Übereinkunft, dass man seine Zeit wohl kaum so klar kategorisieren könne. Dann, von dem, dessen theoretische Grundlagenforschung auch in 100 Jahren noch relevant sein wird (mehr als jetzt): I don’t care about the future.

I do. Aber vielleicht macht das keinen Unterschied.

Tutorials I attended

• General Game Playing is the task to write programs that learn to play arbitrary games solely by being given the rules of a game. Allow games with an infinite number of states and this is as close as you can get to working on AGI without being considered weird by the traditional AI community.
• Autonomous Bidding Agents: If you want people to bid their true values in an auction, use a sealed-bid second-price auction (similar to eBay’s system). The Trading Agent Competition is a useful testbed if you like game theory and view AI as a tool for automated trading and scheduling.
• Constraint-Based Local Search in Comet: If you want to solve constraint satisfaction problems (e.g. a Sudoku), don’t want to spend much time programming and like nice visualizations, use Comet.
• Practical Statisticial Relational AI: We may finally be able to unify logical inference, inductive logic programming, probabilistic inference, and statistical learning using Markov logic networks. Alchemy is supposed to fulfill Prolog’s promises (and it looks like it could).

Talks I heard

• Agents, Bodies, Constraints, Dynamics and Evolution: Robot soccer is a great challenge. We can’t completely avoid ethical choices (but please, don’t think ahead too far, let’s start with Asimov). Robot architectures need to provide an easy way to model constraints on the agent’s actions.
• Graph Identification and Alignment: Nice algorithms for entity resolution, link prediction, and collective classification exist that make it possible to extract useful information from noisy input data, e.g. social relations from a bunch of e-mails.
• AI in a Moore’s Law World: The Stories of Farecast and KnowItAll: The story of Farecast: You can make lots of money using data mining. The story of KnowItAll: It would be awesome if web search engines understood web pages and answered questions instead of just doing keyword searches, but we’re really not there yet and we need much more computing power for more sophisticated approaches.
• Representing and Reasoning about Preferences: You can force people to vote truthfully instead of opportunistically by making manipulation a NP-hard problem.
• Big “A”, Small “I”: Smart Ends from Simple Means: If you are designing a game, don’t compute things the player never gets to see, think about whether sophisticated planning really is better than just-the-next-step computation and remember that Matt Brown likes to do things in a very non-rocket-science kind of way.

Technical sessions

• Deriving a Large-Scale Taxonomy from Wikipedia: Wikipedia’s categories make for a useful network of concepts and, with a little effort, are just as good as the current largest taxonomies, WordNet and ResearchCyc.
• A New Algorithm for Generating Equilibria in Massive Zero-Sum Games: The range of skill in a game, i.e. how many different skill levels exist, is a reasonable measure of the complexity of a game. There is an iterative algorithm for computing approximate equilibrium strategies by fixing the opponent’s set of strategies but I don’t remember how it works.
• Reasoning Patterns of Agents: We can think of five basic reasoning patterns agents use in games — direct effect, influence for no reason, manipulation, signaling and revealing/denying — and these can be used to talk about actions in a more fine-grained way than just saying that an agent maximizes expected utility.
• On the prospects of building a Working Model of the Visual Cortex: More computing power is good and Jeff Hawkins approach may not be totally off, but we don’t want to mention his name.
• Modeling Crowd Behavior using Social Comparison Theory: People act similar to those who are like themselves but not too much like themselves. Simulate this and what you get is fairly convincing crowd behavior.
• Retaliate: Learning Winning Policies in First-Person Shooter Games: Really simple reinforcement learning produces good team strategies for Unreal Tournament’s domination mode.
• Analyzing Reading Behavior by Blog Mining: People who write comments on your blog tend to be regular readers. People who visit your blog are likely to visit similar blogs, too. If you don’t believe this, remember that we can still mention preferential attachment in our paper and thus have a few formulae that make the obvious much more convincing.

(Not quite) random remarks

• Man vs. Machine Poker Tournament: Poker players are lots of fun. This is the last year the human players won, but it is still not clear whether the bot that wins next year will be a boring equilibrium player or a learning bot that exploits its opponent’s weaknesses.
• The outside view of “traditional” AI research is right. I got the impression that most people are happy working on smallish problems. Let’s improve an existing optimization algorithm here and think about a new heuristic there, but don’t even mention general intelligence. That’s science fiction.
• And wrong. Whatever you do, be it natural language processing or robotics, the signs are there that quick hacks won’t get you anywhere near intelligent behavior, that the combination of faster hardware and new neuroscience provides an upper bound for the advent of silicon intelligence and that there are ethical and societal issues that need to be taken care of.
• Times change. On the way back from the conference, an uncle of mine who lives in Vancouver told me about his youth. Most of the time progress feels slow and boring. When you just return from a place where 200 people think about how to make the international network of computers reply to questions in an intelligent way and someone tells you about how he started out as a kind of millwright 50 years ago, that’s not the case.

1010101010101010101010101010101010101010_

Nun habe ich dieselben Zahlen, darf aber annehmen, dass einfache Erklärungen wahrscheinlicher sind als komplizierte. Kann ich jetzt eine Vorhersage treffen? (Nein.)

1010101010101010101010101010101010101010_

Ich nehme zusätzlich an, dass die Folge auf irgendeine Weise berechnet werden kann. Kann ich jetzt die wahrscheinlichste nächste Zahl vorhersagen? (Nein, nicht in endlicher Zeit. Das allgemeine Vorhersageproblem ist unlösbar.)

1010101010101010101010101010101010101010_

Kann ich einen Algorithmus schreiben, dessen Ausgabe gegen die wahrscheinlichste nächste Zahl konvergiert, wenn die Laufzeit gegen geht? (Ja. Dovetailer über alle möglichen Programme, angefangen mit dem kürzesten; die Ausgabe des kürzesten Programms, das die Zahlen und eine zusätzliche ausgibt, ist die aktuelle Hypothese.)

Wir Menschen treffen jeden Tag Vorhersagen. Der Vorteil, den wir gegenüber derzeitigen Algorithmen haben, liegt in den zusätzlichen Annahmen, die wir unbewusst machen, in der Art von inductive bias, mit dem uns die Evolution ausgestattet hat. Die Aufgabe von Forschern auf dem Gebiet des maschinellen Lernens ist es, den Suchraum von Algorithmen derart einzuschränken, dass die Laufzeit der Algorithmen polynomiell wird, und dabei möglichst wenige Lösungen für Vorhersageprobleme auszuschließen, die für unsere Welt relevant sind.

The infinite possibilities each day holds should stagger the mind. The sheer number of experiences I could have is uncountable, breathtaking. And I’m sitting here refreshing my inbox. We live trapped in loops. reliving a few days over and over, and we envision only a handful of paths laid out ahead of us. We see the same things each day, we respond the same way, we think the same thoughts, each day a slight variation on the last, every moment smoothly following the gentle curves of societal norms. We act like if we just get through today, tomororrow our dreams will come back to us. — xkcd

In einer Welt, die ohne vorgegebenen Sinn einfach existiert, erfinden wir uns und das, was wir als sinnvoll ansehen, auf dem Weg in die Zukunft. Jegliche Gründe, Ziele und Zwecke jenseits der biologischen erschaffen wir selbst. Aus “Wie soll ich handeln?” wird “Wer will ich sein?” und “In welcher Welt will ich leben?”.

Mit jeder Handlung machen wir aus dem aktuellen Zustand unserer Welt einen anderen. Der Raum aller möglichen Zustände ist die Menge der Zustände, die keine physikalischen Gesetze verletzen. Denkbar ist eine astronomische Zahl, wünschenswert sind die wenigsten davon. Die Menge der Zustände, die bewusstes Leben enthalten, macht nur eine winzige Ecke im Raum aller möglichen Zustände aus. Unsere Welt ist ein Punkt irgendwo in dieser Ecke.

Manche Zustände unterscheiden sich stärker voneinander als andere. Der Zustand der Welt, in der die Tasse Tee neben mir zwei Zentimeter weiter links steht, liegt näher am aktuellen als der, in dem sich die Tasse in einen feuerspeienden Drachen verwandelt hat. Ein mögliches Maß für den Abstand von zwei Zuständen wäre eine Art Informationsdistanz: Die Länge oder Laufzeit des kürzesten Algorithmus, der aus einer vollständigen Beschreibung von Zustand A die entsprechende Beschreibung von Zustand B berechnet.

Indem ich mich für eine Handlung entscheide, wähle ich einen unserer Nachbarn im im Raum aller möglichen Zustände. Wie sieht die Teilmenge der Zustände aus, die vom jetzigen Zustand der Welt aus durch mein Handeln oder Nicht-Handeln erreicht werden können? Das bestimmt, welchen Einfluss ich mit meinen Entscheidungen als einzelner auf die Welt habe.

Die Frage, was festlegt, welche Zustände ich erreichen kann und welche nicht, führt unmittelbar zu der Frage danach, was unsere Position im Raum aller möglichen Zustände bis jetzt am stärksten verändert hat. Die Antwort definiert, was Optimierungsprozesse sind: Systeme, die den Zustand unserer Welt auf einen kleinen Zielraum mit bestimmten Eigenschaften hin bewegen.

• Evolution ist ein Optimierungsprozess, der Replikatoren — Bakterien, Tiere, Menschen und die Gene dahinter — durch Mutation, Rekombination und Selektion auf effektivere Vermehrung hin optimiert.
• Ein Schachcomputer ist ein Optimierungsprozess, der aus der Vielzahl möglicher Kombinationen von Schachzügen die auswählt, die die Position der Figuren auf einem Schachbrett so verändern, dass sich die Welt in einen Zielraum mit der Eigenschaft “Der Schachcomputer gewinnt.” bewegt.
• Menschliche Intelligenz ist ein mächtiger Optimierungsprozess, der für verschiedenste Ziele eingesetzt werden kann. Rationalität erreicht klar definierte Ziele, nonlineares Handeln die unbewussten.

Cognitive Science ist die Lehre von den Optimierungsprozessen. In Psychologie und Neurobiologie wird der effektivste bekannte Optimierungsprozess, das menschliche Gehirn, analysiert, in Mathe, Informatik, Statistik und Logik werden die methodische Grundlagen für den Bau von künstlichen Optimierungsprozessen unterrichtet.

Optimierung ist ein Vorhersageproblem. Jeder Maschine steht eine festgelegte Menge an Aktionen zur Verfügung. Um ein Ziel zu erreichen, muss die Maschine vorhersagen, welche Kombination aus Aktionen die Welt dem Zielzustand am nächsten bringt. Dass wir Menschen die Auswirkungen unserer Handlungen vorhersagen können, zeigt, dass Quanten- und Chaoseffekte bei Vorhersagen umgangen werden können, wenn man Abstriche bei der Genauigkeit der Prognosen macht.

Intelligenz ist die Fähigkeit, akkurate Vorhersagen zu treffen um Aktionsfolgen zu finden, die unsere Zukunft auf kleine, weit entfernte Regionen im Raum des Möglichen hinsteuern. Die Frage, ob künstliche Intelligenz möglich ist, lautet eigentlich: “Wie weit werden wir uns übertreffen? Wo liegen die Grenzen der Berechenbarkeit?”

Weil die Grenzen, denen wir unterliegen, universell sind, sehen wir sie nicht. Algorithmen, die Information optimal extrahieren, unterliegen keinen kognitiven Fehlern. Die unvoreingenommene Instrumentalisierung aller verfügbaren Mittel stellt einen enormen Machtzuwachs dar; als Menschen schaffen wir es oft nicht, funktionaler Fixiertheit zu entrinnen, sobald wir einmal gelernt haben, wozu etwas gut ist.

Im nächsten und letzten Schritt, der genauso unvermeidbar und unintuitiv ist wie die davor, schreiben wir Optimierungsprozesse, die den Teil ihrer selbst restrukturieren, der für das Optimieren zuständig ist. Algorithmen, die vorhersagen, welche Veränderungen es braucht, um bessere Vorhersagen zu treffen. Prozesse, die Welt auf Zielregionen hin bewegen, von denen wir nicht gedacht hätten, dass sie in unserer unmittelbaren Nachbarschaft liegen.

It is amazing that, after almost 30 years of philosophical discussions, John Searle’s argument against the possibility of programming a robot in a way that makes it really think is still alive. I am now going to analyze what it means for an entity to have intentionality, then give a short account of the strongest version of Searle’s thought experiment and finally argue that the only way to deny intentionality to robots on the grounds of Searle’s thought experiment is to assume a priori that intentionality is tied to biochemical processes.

Intentionality is the difference between a human being that answers questions about a story and a machine that uses a lookup table that includes all possible questions and appropriate answers. The difference between arriving at an answer by understanding the question compared to arriving at an answer by purely mechanical means is to be found not in different behavior but in the mental states that are present in the process of answering the question.

Mental states are conditions like being in pain, having the experience of seeing red, feeling love and remembering that the Statue of Liberty played an important role in a story. A mental state cannot be present without an entity that experiences the state, therefore there can be no understanding without some entity that understands. Since experience cannot be present without consciousness, the entity needs to be conscious. By definition, entities that are not conscious don’t have subjective experiences.

For very simple systems like thermostats, it is safe to assume that no understanding is involved. Even for systems that have or exceed human capacities in a specific realm, e.g. chess computers, we can assume that these systems do not have any intentionality.

Suppose we design a program that does not just represent information by means of symbols that can be identified with real-world counterparts but simulates the actual sequence of neuron firings at the synapses of a human brain. Suppose further that this program is part of brain-shaped computer that is contained within a robot that receives sensory input, e.g. through cameras, and communicates with the world by motor output in a way that is analogous to the motor capabilities we have. If the behavior that this robot shows is indistinguishable from the behavior of real human beings, should we attribute intentional states to the robot?

According to Searle, such a robot cannot have intentionality. A little man inside the robot could be receiving uninterpreted, purely syntactical symbols from the robot’s sensory receptors, he could follow fixed rules that tell him how the input symbols relate to manipulating a system of water pipes that are modeled similar to the neuronal connections of a human brain and how the result of the pipe manipulation is connected to returning other uninterpreted formal symbols to the motor mechanisms of the robot. If the man knew nothing about what the purpose of manipulating the water pipe system was, we would certainly not attribute understanding to the man.

Therefore, Searle argues, we cannot conclude that something has intentionality on the grounds that it has a certain sort of input and output and a program in between. Furthermore, we don’t have any reason to assume that intentionality has anything to do with computer programs, i.e. computational operations on purely formally specified elements, since for any program it is possible for something to instantiate that program and still not have any mental states.

Although the claim that behavior is not sufficient to make attributions of intentionality is true, it is not sufficient to conclude that no program can possibly exist that, when instantiated, has mental states. The systems view of Searle’s robot — even if the man himself has no understanding, the conjunction of man and water pipes does have understanding — sounds less absurd if you take a moment to think about how such a system would look like. In contrast to what Searle claims, it is not possible for the man operating the water pipes to internalize the formal structure of the water pipes since no system can perfectly simulate a system of comparable complexity within itself.

Even if the man could internalize and simulate a system analogous to the 10 trillion neurons of our brain with on average 7,000 synaptic connections to other neurons, it is not clear that the simulation of all the chemical and physical interactions going within the quadrillions of interconnections will not give rise to emergent phenomena, one of which may be actual understanding. It makes a lot of sense to point out, as Searle does, that there is a significant difference between physical phenomena and the simulation of physical phenomena. A simulated rainstorm does not make you wet, a simulated fire does not burn down anything. Why should we suppose that a simulation of understanding, be it executed by a computer or by another brain, actually understands anything? Because mind may be fundamentally different from other phenomena in that it is not so much the physical constituents that make up the phenomenon but the pattern of the constituents. Patterns don’t get lost when they are simulated. We cannot be sure that it is the patterns that matter, but we cannot be sure that this is not the case either.

Searle seems to be sure that this is not the case when he states that, as long as we simulate only the formal structure of the sequence of neuronal firings at the synapses, we are still missing what matters about the brain, that is, its ability to produce intentional states, since even such a low-level simulation is no more than syntactic manipulation that relates symbols to each other but not to the external world. It is obvious that, if you subscribe to the view that only certain biochemical processes can cause experiental states, there is no way to convince you that other physical arrangements may result in the same experiential states. However, it is important to note that Searle’s claim that “all the computer has is more symbols” is not sufficient to justify this view. The information that is received by our sensory receptors and converted into electrical signals is no less symbolic than the information that is processed by the program of the robot. The information is no more grounded in reality than the signals from the light that hits the camera of the robot.

Therefore, to say that no purely formal model will ever be sufficient for intentionality because what matters about brain operation is not the formal shadow cast by the sequence of synapses but rather the actual properties of the sequences can only justified by an a priori assumption that intentional states can be caused by biological mechanisms exclusively. If you take into account the fact that biology, which follows the rules of physics, is no more semantic than computational models which follow purely syntactical rules and that physical events involving complex, large-scale patterns may result in phenomena that are qualitatively different from their low-level consituents, you don’t have much ground for the claim that robots can’t show understanding just because they are implemented as a computer with the right sort of program.

In 1667, when the chemistry behind fire was still a great mystery, Johann Becher theorized that all flammable materials contain phlogiston, a substance without color, odor, taste, or weight, that is liberated in burning. Similarly, two hundred years later, Lord Kelvin went on record saying that “[life's] power of directing the motions of moving particles, in the demonstrated daily miracle of our human free-will, and in the growth of generation after generation of plants from a single seed, are infinitely different from any possible result of the fortuitous concurrence of atoms… Modern biologists were coming once more to the acceptance of something and that was a vital principle.”

Nowadays we know that combustion requires oxygen, a substance that was well known in Becher’s time, and that life is not caused by a mystical essence that makes inanimate substances animate but by cell processes that can be explained without any physics that were not known in Kelvin’s time. We cannot exclude that Searle’s suggestion that understanding can only be caused by a hitherto unknown mechanism in the brain’s biochemistry is true, but it nonetheless strikes me as improbable and may well be a repetition of the common error of making up new laws of physics instead of trying to see where a phenomenon fits into the current framework of the natural sciences.

[1] The entity may not necessarily be physical, but the occurrence of a mental state implies that something experiences the state.

[2] This implies that either animals have some degree of consciousness or that animals are no more than machines without actual intentionality. Cf. Jean-Paul Sartre, Being and Nothingness.

[3] Which, to our knowledge, could be computable.