‘Beginning to reason is like stepping onto an escalator that leads upward and out of sight. Once we take the first step, the distance to be travelled is independent of our will and we cannot know in advance where we shall end.’

— Peter Singer (1982)

‘You seriously believe a universe in which billions of sentient beings on this planet alone die horrible deaths in war, famine and plauge [sic], a universe in which people have a fleetingly short time to live before their health and strength drain away, a universe in which misunderstanding is endemic and barriers between minds are unbreechable, a universe in which most sentients are forced to serve others to obtain the basic necessities of life and in which a great many live under fear and repression, a universe where the only known planet with sentient life is under constant threat of being wiped clean by a whole range of disasters, where the entire population is under some degree of harmful delusion, doesn’t suck?’

— Michael Wilson

There are many situations where we lack knowledge to move the world from an undesired state to a more desired state. We call these situations problems. In the following, I describe what I see as one of the major challenges that a technical solution to the problem of problem solving faces.

If we cannot ignore a problem, what we do is this: We take our limited knowledge about the world, our limited knowledge about what we want, and use our limited reasoning capabilities to find out what we should do to move the world closer to how we want it to be. Given that we believe the world to be like this, and given that we would like the world to be like that, we infer what action we should take.

In general, inference denotes the process of assuming that certain statements about the world are true and deriving what follows for the truth of other statements. Machines are potentially much better at inference than the human mind is. Programs have the potential to encode much more and much more precise knowledge than any one of us could learn in a lifetime. The laws of probability theory and approximations thereof can be used to infer precise knowledge about the world from data and to reason using this knowledge.

The fact that we can use machine inference to solve problems that are too difficult to be solved using human reasoning makes research into inference methods important. When we use machine inference to figure out what the dynamics of protein folding are in order to solve diseases like Alzheimer’s, we do so because, given all our data and our wish to cure the disease, our human minds still cannot figure out the cure on their own. A prerequisite for the use of machine inference is to have the problem statement available formally, as a program, a mathematical object. By singling out a small, well-defined problem, we can formally write down knowledge about the problem domain (or a program for inferring such knowledge from data) and a program that uses this knowledge to solve a particular reasoning task, i.e. to help us in determining how to change things for the better.

Any such small, formal problem statement captures only very little about what we want things to be like more generally. What do we want things to be like? We know that there are some things we want because they lead to other good things — these we call instrumental values — and there are some we want for their own sake — terminal values. We can make guesses about what our terminal values are, what we value for its own sake — joy, freedom, discovery, beauty, kindness — but ultimately, what we want the world to be like is not summed up well by any (or all) of the individual concepts behind our guesses. The name for that which does capture all about what we want is preference.

Because preference is stored opaquely in our brains, we cannot directly access its content, we can only use it to some extent. Similarly, we have little access to how our minds represent concepts like “tree”, “word”, or “spring”. This does not mean that there is no precise structure behind any given concept. On the contrary, the theories that best predict human concept learning and reasoning in recent psychological experiments are those that assume that concepts are represented as probabilistic programs.

Likewise in the case of preference: Our meager introspective abilities obscure the fact that the term ‘preference’ denotes a precise informational structure. This is easy to see where limitations of reasoning make us more uncertain than dictated by what can be deduced from the information that we know is stored in our brains, but is likely to be true more generally.

Take our search for a cure for Alzheimer’s: We have an intuitive idea what results are good and what results are bad — those that actually cure the disease and that do so without side-effects are the good ones. Nonetheless, we are uncertain about how our preferences rank different states of the world; without using machine inference to figure out the dynamics of protein folding, we do not know how preference orders possible states of the world because we cannot tell which state corresponds to a cure and which to a useless substance. By using machine inference to determine which state corresponds to a cure and which does not, we factor out a small part of our preferences that we assume to be independent from the rest (although it is not!). We thus hope to improve our understanding of what this part of our preferences says about the world.

When we — or our machines — work towards the solution of any particular subproblem without taking into account our preferences as a whole, we commit what could be called a mistaken factorization of preference. Preference as a whole makes a statement about what is the best choice at any given point in time, and if we look at only a small part of this statement, we lose value. On the other hand, if we can access the formal statement that our preferences make as a whole, then we may be able to use the reasoning capabilities of machines to determine much more precisely what the best choice looks like than would be possible through introspection.

If preference is a mathematical structure — even if it is currently implemented in our brains in a distributed and implicit way — then what kind of structure could it be? I do not know the answer to this question, but there are situations that are similar in the sense that they also take an intuitive idea and reify it into a mathematical object.

In computer science, there is the notion of the future of a computation. For example, at point # in the program (* 3 (+ 4 #)), what the future holds is that whatever value we hand it, it will add 4, multiply the result by 3, and then do whatever it does to the return value of a program that has finished, e.g. print the result to the screen. The notion of a continuation captures the idea of taking the future of a computation and storing it in an object. If we capture the continuation at #, the future starting from this point becomes a mathematical object, a value that we can pass around just like any other object and that we can reason about formally.

Analogously, we would like to take the diffuse notion of the preferences of a decision-making system (like you and me) and reify it into a formal object. And analogously, we expect this object to be a computational structure that contains information about what will — or, in the case of preference, should — be done in the future, but it may take lots of computation to determine what exactly this information says.

The project of formalizing preference has two parts: understanding the structure of preference (i.e. what kind of object are preferences, how do they compose) and getting at the actual content of human preference (i.e. extracting or pointing to the preferences of a given agent).

There are proposals for what the structure of preference could look like (e.g. preference logics, utility theory), but they seem insufficient in non-trivial situations. Two examples of such situations are (1) that we want our preferences not to lose meaning when it turns out that we have been mistaken about some of the things our preferences talk about (the so-called ontology problem), and (2) that we may have preferences about how we want preferences to interact. Different people appear to want different things, and even within a single mind, seemingly contradicting wishes exist. For example, there are things we want to do and there are things we actually enjoy doing, and these are often not the same. How do we figure out the statement that such a system of preferences makes about what should be done? This is called preference aggregation across agents and could be called compositionality of preference within a single agent.

Formalizing the content of our preferences, i.e. pointing to preference in a precise, machine-readable way, poses similarly challenging problems. The strongest illustration of this that I can currently think of is the following: If our preferences determine which method of formalizing their content is the correct one (namely the one that results in our actual preferences), and if we cannot know or use our preferences with precision until they are available as mathematical objects, then how can we find the correct formalization method? I can imagine that knowing the structure of preference would clarify what properties a method needs to have that allows us to formally refer to the preference content of any given agent, but to what extent this is the case is an open question.

To summarize, the fundamental problem is this: We have only limited access to what we want, and we cannot really figure out what follows from that which we do know about what we want. Machines are potentially much better at reasoning about what follows if we can give a formal description of what we want. However, if we formalize only a few small problems, we lose value due to our limited reasoning about the remaining part of our preferences and due to assuming independence between preferences when in reality they are intertwined. We need to understand preference as a formal object if we want to use machine inference to figure out what should be done to make this world a nicer place.

I thank Vladimir Nesov for useful discussion and for originating many of the ideas mentioned here.

Das, was uns durchgedacht und zugeschnürt vorgesetzt wird, mögen wir akzeptieren, aber wir werden es niemals verteidigen. Null Inferenzschritte. Nur das, was wir selbst entdecken, machen wir uns zu eigen. Wenn wir bereits Ideen absorbieren, für die wir uns ohne Überzeugung und von der Endidee ausgehend Argumente ausdenken, wie viel stärker fühlen wir uns dann zu Ideen hingezogen, die wir selbst erdacht haben?

Wissenschaft funktioniert, weil jede Veröffentlichung (hoffentlich) Daten enthält, von denen aus wir den letzten Inferenzschritt selbst vollziehen können. Kunst funktioniert, weil sie Ideen nimmt und von dort aus einige Inferenzschritte rückwärts geht.

In einer Welt idealer Rationalisten macht es keinen* Unterschied, ob die letzten gedanklichen Schritte selbst ausgeführt oder fertig präsentiert werden. In unserer Welt dagegen ist es leicht, mich von einer Idee zu überzeugen. Ich muss die Idee dazu nur als meine eigene ansehen.

I just wrote my first letter in ten years and it felt strange. The blue liquid flowing out of my pen and onto the thin sheet of cellulose in front of me. The cell walls of a dead tree, now functioning as a kind of disposable monitor. The paper soaked with watery circles and lines, clearly one of the most wasteful ways to store one kilobyte of plain text. In a few minutes, on my way to the Christmas market, I will put this unlikely storage medium in a yellow box next to the sidewalk, knowing that tomorrow, someone will pick it up, drive it across Germany and bring it not to the addressee, but to yet another box where she will show up, sooner or later. This takes roughly 100.000 times as long as an e-mail.

E-mail is less awkward, but not by far. Part of me enjoys typing really fast, probably due to having seen too many hacker movies in my teenage years. The rest of me snickers at the idea of moving muscles and bones, pushing fingertips on black plastic, in order to transmit information from one system using electrical signals to another one. For each bit that makes its way from my head into my computer, I move a billion billion billion electrons when one would suffice.

Each intermediate step in the process of information transmission creates borders between us and makes our conversations less intimate. Bandwidth is growing, delays and barriers are going away (the final barrier being the conversion from semantics to syntax and back).

In 2007, writing a letter is like playing with mud and electricity because you are hungry and it can’t take that long until something akin to an apple tree evolves.
 
Like taking money out of your bank account and giving it away minutes later in exchange for the thing you really wanted even if you could have paid with your EC card, because you always did it this way.
 
Like taking pictures with your old analog camera and scanning them later on, because style is not defined in pixels per cm².
 
Like writing a letter, because the textual content was little more than an envelope, because what you actually said was “I care”, and because the most efficient way would have been the least effective.

What appears to be context may be information, what appears to be information may be context. The failure or refusal to accept the unspoken social contract that defines which is which is one of the main reasons why nerds are socially inept. Think small talk.

The problem with informal speculation is that it is easy to be unclear in your writing, and being unclear in your writing usually results from being unclear in your thinking.

Take, for example, my last post, where I was speculating about the properties of recursively improving systems. When I wrote that a system cannot predict a system of greater algorithmic complexity, I did not make clear whether I meant that not all systems of a certain greater complexity can be predicted (which is true) or that no system of a certain greater complexity can be predicted (which is false). For a system to improve recursively with respect to some goal in a way that increases its complexity, there is no need for it to be able to predict all systems of a certain greater complexity. Thus, the whole argument breaks down.

The problem with formal argumentation is that, even if you resort only to the most basic rules of logic, what you prove might not be what you intended to prove.

Take, for example, the paragraph above. The claim that some systems can learn to predict systems of higher algorithmic complexity can be proven formally. You define what you mean by “system”, “complexity” and “learn to predict” in mathematical terms, show an example of two systems, one with lower, one with higher algorithmic complexity, and how the former can learn to predict the latter. From now on, you are free to proclaim that there are simple systems that can learn to predict complex systems. Impressive!

Caveat: Do not mention that you were using the standard definition of “learning to predict” which says that a system learns to predict another system if, after a finite number of observations, the system knows all following outputs of the other system. And, please, stay quiet about the fact that the system that was predicted in your proof did not output anything but zeros after a finite time of complex behavior. Otherwise, people might think that what you have shown has little relation to what is usually meant when we talk about “learning to predict” behavior. And, more destroyingly, they would be right.

As soon as the context changes just a little, as soon your assumptions differ just a little, the value of a formal argument immediately becomes negative. Not only does such an argument say nothing about whether a conclusion is true or false, it will also let you sleep soundly, with the security that there is no need to further think about what you know — it is proven.

Informal arguments, on the other side, cannot provide security in the first place. The truth of informal arguments depends on what you mean by the words you use. Different people associate different meanings with different words, and what was once a discussion soon becomes a game for idle linguists — a fact that is painfully clear if you are doing philosophy. When rational people disagree, even after prolonged discussion, you can almost always trace it back to words being used in slightly different ways.

Your blurry, informal argument based on theorems used out of scope might well convince me. At that point, arguing has long stopped being our joint search for truth. Why bother?

(I don’t do programming anymore.)

Wir Menschen unterscheiden uns nicht großartig in unseren Wünschen. Wir wollen Glück, Freude, Freiheit, Unabhängigkeit, Sicherheit, Wissen, Kreativität, Individualität, Sexualität, Freundschaft und Liebe (nun ja, Männer zumindest). Wir schätzen unser Leben, das unserer Freunde, unserer Familie und das unserer sechs Milliarden Mitmenschen. Trotzdem ziehen wir in verschiedene Richtungen, konkurrieren, intrigieren und machen generell den Eindruck, als ob wir es darauf anlegen, paradox zu handeln.

Wenn wir verstehen, welches Ausmaß die Zukunft hat, die auf dem Spiel steht, und wenn wir uns im Großen und Ganzen einig sind, was uns jetzt und für diese Zukunft wichtig ist, warum funktioniert es dann nicht besser^TM?

Warum leben wir nicht längst in Utopia, wenigstens asymptotisch?

Die Erklärung, die ich nicht glaube: Es geht nicht besser. Würde man jeden Menschen fragen, wie sehr diese Welt seinen Vorstellungen entspricht, und so zu einem Gesamtbild kommen, so gäbe es nichts, was dieses Bild dauerhaft besser machen könnte. Für diese Erklärung spricht die Anpassungsfähigkeit unseres Gehirns, die daran schuld ist, dass die meisten Änderungen unsere Gesamtzufriedenheit nicht dauerhaft verbessern. Glück ist die erste Ableitung positiver Veränderung. Aber, erstens: Lasst uns die offensichtlichen Unmenschlichkeiten dieser Welt beheben, dann können wir noch einmal darüber reden, ob es nicht besser geht. Zweitens: Manche Leute scheinen immer ein bisschen glücklicher zu sein als andere. Gene und Umwelteinflüsse legen die Biochemie unseres Gehirns fest und wir sind dabei, beides zu verstehen.

Die Erklärung, die ich gerne glauben würde: Die Probleme unserer Welt sind komplex. Wir sind auf dem Weg zu Lösungen, aber die erfordern ein gewisses Mindestmaß an Zeit und Technologie. Es wäre falsch, sich an neue Technologien zu klammern, weil diese beinahe immer zu polaren Zwecken eingesetzt werden können, aber ein Blick auf die Geschichte macht klar, dass neue Technologien Einfluss haben. Die Kombination aus omnipräsentem mobilem Web für die Massen und Suchmaschinen, die natürliche Sprache verstehen, könnte die Wissensverteilung weiter demokratisieren. Prognosemärkte (die von Google, Microsoft, HP und Intel bereits intern eingesetzt werden) könnten Teile der Politik rationaler gestalten, der Anfang der vollständigen Aufzeichnung der Menschheitsgeschichte alle kollektiven Entscheidungen.

Die Erklärung, die immer nur andere betrifft: Das sind alles egoistische Nichtsnutze, denen die Menschheit egal ist, so lange sie Familie, Job und ein halbwegs interessantes Leben haben. Unterstützt werden sie in ihrer Haltung von Wissenschaft und Wirtschaft, die Gedanken über den Lauf der Welt zugunsten kurzfristiger und handfester Resultate bestrafen. Andererseits werden gesellschaftliche Fragen gerne mal eben beim Mittagessen gelöst (wenn gerade keine Fußball-WM stattfindet) und mit zufriedenem “Tja, so müsste man’s machen” abgehakt. Zu Handlungen kommt es natürlich nicht, denn dafür bräuchte man Lösungen, die tatsächlich funktionieren, müsste herausfinden, wie man als einzelner zur Umsetzung beitragen kann, und müsste die Lösungen finden, von denen man selbst profitiert. Wozu die Menschheit retten, wenn es nicht entweder Geld, Sex oder Status bringt oder sowieso auf dem Weg zur Rettung des eigenen Lebens liegt?

Die Erklärung, die mich (und dich!) betrifft: Wir arbeiten auf Teilziele hin, die nicht direkt dem entsprechen, was wir wirklich wollen. Weil das fast jeder tut, weil verschiedene Teilziele oft gegensätzliche Aktionen erfordern und weil die Ziele selbst dann oft nicht erreicht werden, heben sich unsere Bemühungen mehr oder weniger auf. Unser Tun führt so zwar zu neuen Methoden und zu neuen Erkenntnissen über unsere Welt, die indirekt zur Realisierung unserer Wünsche beitragen können, ist aber ineffektiv und potentiell schädlich. In dem Moment, in dem wir uns einer Ideologie verschreiben, weil wir glauben, dass die Durchsetzung von deren Axiomen den Menschen das geben wird, was sie wirklich wollen, arbeiten wir an der Verbreitung der Ideologie und nicht mehr an den eigentlichen Problemen.

Glücklicherweise ist die Lösung einfach: Wir wählen in jedem Moment die Handlung, die für sich genommen am ehesten unseren Werten entspricht, anstatt uns auf eine Ideologie oder auf ein langfristiges Ziel festzulegen und darauf hinzuarbeiten.

Dummerweise funktioniert sie nicht in jedem Fall, insbesondere dann nicht, wenn wir existentielle Risiken — Katastrophen, die das Ende der Menschheit bedeuten können — in Betracht ziehen und uns der Fortbestand der Menschheit doch ein bisschen kümmert.

KI in zwei Sätzen: Die Annahme, dass wir in absehbarer Zeit auf einen relativ allgemeinen Mustererkennungsalgorithmus stoßen, der mit genügend Rechenpower die Mustererkennungs- und Vorhersagefähigkeiten des menschlichen Gehirns übertrifft, ist (für diese Art von Annahmen) weit verbreitet. Deutlich kontroverser ist die Idee, dass Algorithmen praktisch möglich sein könnten, die Veränderungen an sich selbst vornehmen, um so große Klassen von formalisierbaren Probleme bestmöglich zu lösen — unabhängig davon, wie anspruchsvoll diese Probleme sind, d.h. wie viel Intelligenz zu deren Lösung nötig ist.

Die formale Analyse der Approximierbarkeit theoretischer Modelle von Superintelligenz in unserer physikalischen Welt benötigt unsere Aufmerksamkeit, wenn wir wissen wollen, wo auf unserer Liste existentieller Risiken und Chancen maschinelles Lernen steht. Forschung auf dem Gebiet ist ein langfristiges Vorhaben, eines, das jahrelanges Lernen voraussetzt und das mit signifikanter Wahrscheinlichkeit fehlschlägt. Das ändert nichts daran, dass solche Forschung wirklich, wirklich wichtig ist.

Letzte Woche, bei Pasta und Pizza, hat Jürgen die Frage in die Runde geworfen, wie groß denn der Anteil unserer Zeit sei, den wir für das Jetzt leben, und wie groß der, den wir für die Zukunft leben. Zunächst allgemeine Übereinkunft, dass man seine Zeit wohl kaum so klar kategorisieren könne. Dann, von dem, dessen theoretische Grundlagenforschung auch in 100 Jahren noch relevant sein wird (mehr als jetzt): I don’t care about the future.

I do. Aber vielleicht macht das keinen Unterschied.

Auf der Suche nach Wahrheit sollten wir Welt- und Menschenbilder in dem Moment verwerfen, in dem wir merken, dass es uns schwer fällt, Hinweise dafür zu finden, dass sie die Welt besser beschreiben als konkurrierende Ideen. Weil das, was wir momentan glauben, beeinflusst, mit wem wir zu tun haben, welche Informationen wir an uns heran lassen und wie wir Nachrichten interpretieren, sollten wir dem unguten Gefühl im großen Denkknäuel umso mehr Aufmerksamkeit schenken.

Die Realität ist, wie sie ist, unabhängig davon, was wir glauben (und das ändert sich auch nicht dadurch, dass die Autorin eines der populärsten amerikanischen Bücher der letzten Monate das Gegenteil behauptet). Die meisten von uns können nicht vermeiden, im Laufe ihres Lebens der Wahrheit näher zu kommen, sei es in Bezug auf Wissenschaft, Beziehungen oder Menschen im Allgemeinen. Insofern ist es sinnvoller, wenn wir falsche Einstellungen nicht zunächst verteidigen und uns langsam zurückziehen, sondern mit jedem gegenteiligen Indiz die Wahrscheinlichkeit unserer Ideen nach unten korrigieren und unplausible Theorien so früh wie möglich verwerfen. Wenn das bedeutet, zu verstehen, dass Zynismus in manch ungemütlicher Hinsicht Realismus ist, müssen wir auch das akzeptieren (oder private Inseln schaffen, auf denen andere Regeln gelten — aber dem Willen von Welt und Evolution widersetzt man sich nicht leicht und selten auf Dauer).

Menschen sind zu anpassungsfähig, als dass uns kurzfristiges Unglück abschrecken sollte, insbesondere nicht dann, wenn es auf längere Sicht zu mehr Wahrheit und damit zu einer höheren Chance darauf führt, unsere Ziele zu erreichen. Wir gewöhnen uns an praktisch jede Änderung so weit, dass unsere Lebenszufriedenheit nach einiger Zeit der vor der Veränderung entspricht. Wir gewöhnen uns an Klassen von Veränderungen, egal ob Tod (nun ja, zumindest an den anderer Leute — beim eigenen bleibt wenig Gewöhnungszeit), Trennung oder andere Traumata, indem wir abrufbare Verhaltensmuster entwickeln und vermutlich gewöhnen wir uns auch an den Gesamtpegel an Veränderungen in unserem Leben.

In den Augenblicken, in denen wir die Götter nicht dafür verfluchen, dass gerade diese Anpassungsfähigkeit uns gegenüber den Tragödien dieser Welt blind macht, sollten wir ihnen danken, denn manche Veränderungen sind endgültig. Manche Dinge kann man nur einmal sagen und so meinen. Das rettet unser Handeln vor Bedeutungslosigkeit; der Wert dessen, was wir tun, liegt in den Dingen, auf die wir dafür verzichten. Dinge, für die man nichts aufgeben würde, sind nichts wert. Es ist nicht die Hochzeitszeremonie, die dem “Ja, ich will” so viel Wert verleiht, sondern das Wissen, dass wir mit den Worten manche Freiheiten für jemand anderen und für immer aufgeben (oder, wenn wir sie wiedererlangen wollen, das nur unter mittelschweren gesellschaftlichen Strafen tun können).

Was wahr ist ändert sich nicht dadurch, dass wir anderer Meinung sind oder dadurch, dass wir es ignorieren. Das, was wir tun, wird durch das bedeutungsvoll, was wir nicht tun. Glück braucht Bedeutung um nicht leer zu sein, Bedeutung braucht Wahrheit um überhaupt zu existieren. Sowohl Glück als auch Unglück sind Teil unserer Welt und je besser wir diese Welt verstehen, desto mehr Einfluss haben wir auf sie.

Dennett introduces three levels of abstraction we can use to describe intentional systems: Descriptions on the level of physics, a functional perspective and the intentional stance. In order to determine in how far these levels presuppose optimal design and rational behavior, I am first going to explain what Dennett means when he talks about intentional systems, then describe each of the three different levels of abstraction and finally analyze what role the notions of optimality and rationality play for each of them.

When Dennett talks about intentionality, what he refers to is a property of linguistic entities. An entity is intentional if it is not possible to replace part of its description with terms that describe the same physical object and not to lose part of what the entity means. Intentional systems are systems with states that “can be directed on something”, systems that have states that are “about something”.

Obviously, the most basic perspective of looking at a system in the physical world, be it intentional or not, is the physical stance. We base our description of a system solely on the actual physical state of a system and predict its behavior exclusively by applying our knowledge about the laws of nature to this state. Such a low-level stance makes it possible to achieve a level of perfection in our predictions that no other stance allows — only the physical point of view allows us to know beforehand when a system will break down and stop functioning as intended. There is no need to know anything about how close the behavior of the system is to optimality or whether it makes sense to even think about the system in terms of rationality. On the other hand, such a stance can’t tell us much about systems that are not really simple or simplified. Even in predicting breakdown, the causes have to be easily locatable, otherwise the sheer complexity of the system we want to explain will make it impossible to predict anything from a physical point of view.

As systems get more complex, it makes more and more sense to adapt the design stance. Thinking about how a system is designed and which functions it fulfills enables us to predict behavior in a way that is computationally much more economical. There is no need to know how the electrons within the central processing unit of my computer move in order for me to tell that typing a letter will make it appear on the screen. There is just one condition: My computer must not break down this very second, the system we are looking at needs to work as designed. Just like any other predictions on a non-physical level, design stance predictions can’t foresee malfunctions since everything we take into account when thinking about the behavior of the system is the functional design — there is no need to assume that the actual physical condition is part of our knowledge about the system. Every system can be divided into different smaller or larger functional parts. In order to make functional predictions, the first thing one needs to do is to establish the level of abstraction of the individual functional parts. How small are the functional elements that we need to look at in order to make meaningful predictions? For really complex systems, there is no clear-cut answer to this question. Since the human brain is said to be the most complex system in the known universe, we might need to adapt a different, more coarse stance.

Looking at a system from the intentional stance implies that we ascribe to the system the possession of certain information, assume that the system acts in the direction of certain goals and that it always follows the most reasonable action relative to the information and goals the system has. Call the information possessed by the system “beliefs”, call the goals “desires” and the level of abstraction of the intentional stance matches the level we use when we talk about behavior in our everyday life. There is no need to assume that a system that possesses information really believes, but it is nonetheless striking how well these two notions match. The intentional stance not only assumes that a system will function as designed as does the design stance, but also that the design is optimal and that the system therefore always chooses the most rational action relative to its goals. Therefore, whenever we can suppose that it is valid to assume that a system has optimal design, we are justified in assuming the intentional stance as it may well be impossible to apply either the design or the physical stance in a computationally feasible way. Natural selection is an optimization process that molds systems in the direction of optimal design. When we try to find out how close a system is to being optimally designed, we need to take into account the extent of the information the system has and how its actions are constrained.

If a system turns out to be unpredictable from an intentional stance, e.g. because it acts in a way that defies all rules of rationality and logic, we can still collect data about the response pattern of the system and adapt a design stance once we have collected enough data to draw some conclusions about the functional design of the system. Dennett himself makes clear that one can’t stay one the level of “belief” in order to explain the “actual, empirical” behavior of a believer — one needs to descend one level and look at the believer from a functional perspective.

According to Dennett, rationality is optimal design relative to a goal or to a hierarchy of goals and a set of constraints. When we predict intentional systems, what we are doing is no more than to predict what the most rational system would do, given the same goals, constraints and the same information about the state of affairs as the system we want to predict. The assumption of rationality entails a few other beliefs about the system that we suppose is rational. We can assume that, usually, it does not desire its own destruction and that it follows the truths of logic, since assuming that the system has some beliefs but acts contrary to what follows from these beliefs would violate basic rules of rationality. Systems in the real world are not perfectly rational, therefore not all logical truths appear in those systems, but, in order to make predictions from an intentional stance, we just assume perfect rationality and hope that the drawbacks are smaller than those associated with any other stance.

Dennett introduces the concept of an intentional system in order to connect the intentional domain to the non-intentional domain of the physical sciences. It is quite obvious that he achieves this goal. From a pragmatic point of view, our beliefs can be identified with the information we possess about the world, our desires are our goals and a large part of our behavior can be described by assuming that we approximate an optimal rational agent trying to achieve its goals within certain informational and physical constraints, a description that completely eliminates intentional wording. Dennett’s article does not explain intelligence, and neither does Dennett aim at a complete theory of mind with this article. What he does talk about is a way of thinking about systems, both intelligent and non-intelligent, that eliminates non-scientific vocabulary and that, in the end, might allow us to talk about and measure intelligence in a way that is both more formal and much more general than the Turing test or any known IQ test.

Andererseits hat jede wissenschaftliche Revolution in den Köpfen einzelner Personen angefangen und sich oft nur langsam ausgebreitet. Die Tatsache, dass eine Idee unpopulär ist, sagt nichts über ihren Wahrheitsgehalt aus. Es gab eine Zeit, als die vorherrschende Meinung war, dass die Erde eine Scheibe sei. Ein großer Teil der in unserer Gesellschaft herumschwirrenden Ideen kann einfach nicht wahr sein — allein aus dem Grund, dass sie sich gegenseitig widersprechen.

Nicht eine Person ist Anhänger einer Idee, die sie für falsch hält. Niemand will einer Randgruppe angehören, die ihrer Version der Realität hinterherläuft. Niemand will Irrtümer und irrationalen Handlungsweisen übernehmen, nur weil sie weit verbreitet sind. Rationales Denken, evolutionäre Psychologie und Entscheidungstheorie können einige offensichtliche Fehler eliminieren, aber Sicherheit gibt es nicht.

Seine Meinung nicht zu äußern hat den Vorteil, dass man nicht so schnell Gefahr läuft, an einer falschen Einstellung festzuhalten, weil das Ändern der Meinung ohne “Ich habe Blödsinn geredet”-Eingeständnisse leichter ist. Und den Nachteil, dass man nicht merkt, wenn man Blödsinn denkt.

Wer die Wahl zwischen zwei Weltbildern hat, wird sich das Weltbild zu eigen machen, das es ihm erlaubt, so weiterzuleben wie bisher und sich dabei gut zu fühlen.

Irrational behaviour is what makes us human. Do you agree? At first sight, scientific evidence seems to point in this direction. Research has shown that bees behave more rational than the average human, that is, they are better at maximizing expected utility. Evolution has done a pretty good job at stripping all the unnecessary and costly irrationalities from a bee’s mind. If a bee’s genes allow it to choose anything but the quickest and most energy efficient way, its hive won’t be the one to survive. As humans, we like to think of ourselves as way beyond the lowlands of evolution. We aren’t forced to behave in a way that maximizes expected utility all the time in order to survive. Are we better off doing it nonetheless?

The bees’ seemingly superior ability to act rational doesn’t tell us the whole story. A bee might in fact be better at fulfilling its desires than a human, but there is more to rationality than optimal goal achievement. The latter is usually termed “thin rationality” or “instrumental rationality”. Should we act according to the principles of thin rationality — always? I’d argue we should. Thin rationality, in fact, doesn’t mean more than achieving your goals in an optimal way. If my goal was to live a joyful life, why wouldn’t I do that in an optimal way? If my goal was to help others live joyful lives, shouldn’t I better pursue it rationally?

Up to now, the most important — and most distinctively human — part of the whole concept of rationality has been left out. There is thin rationality, and there is broad rationality. Remember the rational bees? According to the principles of broad rationality, they aren’t rational. Not at all. Broad rationality is the basis for our conscious choice of goals. How many bees are there which choose to change their goals? But then, how is it possible that we as humans are able to evaluate and change our goals, to choose our destiny?

In order to understand, we’ll have to look at the way our goals are structured. A few seconds of introspection and you’ll realize that there is more than one kind of goal. Psychologists often speak of first-order-desires and second-order-desires. A first-order-desire is, for instance, hunger, which could be expressed as “I’d like to eat”. An example for a second-order-desire would be an aversion regarding the desire to eat (maybe you’re on a diet), in words: “I’d like to not like to eat”.

The concept of broad rationality describes the process of evaluating both your first-order-desires and your second-order-desires, thinking about which ones deserve to be fulfilled and achieving integration of both. The fact that both your first-order-desires and your second-order-desires are likely to be influenced by forces outside yourself makes it quite difficult to decide in favor of one of these two.

First-order-desires are often caused by your genes. Your genes have only one “goal”: To replicate. It doesn’t matter whether you as the “vehicle” will profit or suffer in the course of their “plan”. Second-order-desires are prone to be the victim of another kind of replicator, the memes. A meme is any kind of belief or idea that has been or is currently part of our culture, e.g. “being thin is being beautiful”. What applies to genes, applies to memes, too: Some of them don’t act in your best interest, you as a “vehicle” are only a means to contribute to their replication.

What is so special about humans, after all? It is certainly not the “ability” to behave irrational. Irrational behaviour (in the broad meaning of the word) will lead to the replication of genes and memes which are harmful to the host — that is you. The ability to determine which of your desires are truly yours and to separate them from all the — against your will — meme- and gene-induced goals is special about us humans. Maybe we should additionally learn a lesson from the bees and make use of the principles of thin rationality to act in favor of our goals. Our own goals.