You probably heard of the idea that, at some point in time, we might create systems that solve certain tasks and that get better at these tasks by recursively modifying their code. Here is some scary reasoning:

1. A system cannot predict (=understand) a system of greater algorithmic complexity.
2. Therefore, the only way for a system to improve in a way that increases its algorithmic complexity is trial and error, thereby keeping the best results — i.e. evolution.
3. The only goal that is stable under evolution is rapid reproduction.
4. Therefore, the only stable goal for recursively self-improving systems is rapid reproduction.

I really hope that someone will point out the flaw in this line of thought or show me the reason why it does not apply to our world and to any self-modifying systems we might create.