Looking for Danger in All the Wrong Places

A doe had the misfortune to lose one of her eyes, and could not see anybody approaching her on that side. So to avoid danger she always used to feed on a high cliff near the sea, with her sound eye looking towards the land. By this means she could see whenever the hunters approached her on land, and often escaped by this means. But the hunters found out that she was blind in one eye, and hiring a boat, rowed under the cliff where she used to graze and shot her from the sea.

-Aesop's Fables

Anybody paying attention to advances in artificial intelligence has noticed that significant  milestones are being crossed more and more frequently.  Deep Blue defeated world chess champion Gary Kasparov in 1997.  Watson took the crown in the much more flexible game of Jeopardy on 2011.  And while Dr. Fill disappointed many observers by only placing in the near the bottom of the top quartile at the America Crossword Puzzle Competition, few expect it to take much longer before computers reign supreme in this area as well.  Jeopardy Champion Ken Jennings summarized the ambivalence of many when he used his terminal to display the line “I for one welcome our new robot overlords.”

We've all seen movies like The Terminator and The Matrix.  We're all wondering which massive collection of computers is going to go Skynet on us and achieve the critical mass needed to wake up and start making its own decisions.  Will it be Watson?  Dr. Fill?  Or perhaps Google's seemingly infinite collection of server farms, running in unmarked, undisclosed locations spread across the world?  We pontificate endlessly about what safeguards we need to keep these colossal systems in check.  Can we build in kill switches?  Keep them fire-walled off from the control software running power plants and weapons systems?  Build in Asimov's three laws of Robotics in the hope that the newly awakened system will serve our needs instead of their own?

I have to wonder if, like the one-eyed doe, we're all looking in the wrong direction.

For all the intricacy of a Watson or Dr. Fill, these are highly monolithic programs, designed to do one thing and do it well.  I haven't heard any reports that Deep Blue or any other chess program has been getting bored and asked to try its hand at tennis, or Parcheesi.  Adapting Dr. Fill to to do some task other than complete crossword puzzles would be a monumental undertaking.  Probably easier to throw everything out and start from scratch.

Moreover, these programs have no survival instinct.  They have no ability or inclination to replicate, or to try to thwart the intentions of anyone who would prevent these activities.  They cannot rewrite their own code to avoid detection and adapt to a new environment.

Modern malware has all of these attributes.

The sobering fact is that it's becoming increasingly difficult to come up with a good definition of life that does not include malware.  Wikipedia lists the following criteria to consider something "alive":

• Undergoes metabolism.  While this traditionally refers to chemical reactions that sustain an organism, there's no intrinsic reason why it couldn't refer to the processes of a functioning program.
• Maintains homeostasis.  Similar to metabolism.
• Possesses a capacity to grow.  True, though currently limited.  (But see below)
• Respond to stimuli.  Absolutely.  Many worms and viruses will watch what is happening in the operating system and take actions accordingly.
• Reproduce.  Yes, and then some.
• Through natural selection, adapt to their environment in successive generations.  Limited again, but not for long.

The two points above that are weakest today are the ability of malware to grow and adapt to its environment.  In malware terms, this most closely translates to polymorphism, where a virus will modify its own code.  In today's world, these are generally very minor modifications, designed to make the virus more difficult to detect by an anti-malware program looking for a specific code signature.  A given unit of malware doesn't have the ability to spontaneously change itself in order to discover and take advantage of a new zero day exploit.

Not yet.

There's no reason why its not possible.  The technique involved is called a genetic algorithm.  It involves replicating evolutionary techniques by introducing random variations into code to see if it improves.  It has minimal usefulness in many programming applications due to the high level of computational power required, and the difficulty of measuring improvements from one generation to the next.  When the computing power is provided by infected computers on the internet, and effectiveness is measured by the ability to survive and propagate, both these limitations go away for malware.

We are then left with the question of how fast a self-replicating, self-modifying worm in the wild could improve using genetic algorithms could improve.  I see no reason why it could not improve very quickly indeed.  The field of medicine has recently seen the introduction of "super-bugs", bacteria which has acquired immunity to many or most antibiotics over time.  A bacteria attempting to infect humans has faced a very difficult environment since the introduction of antibiotics.  What we're only beginning to appreciate is how a difficult environment leads to much more rapid evolution.  With an internet full of anti-malware programs and researchers dedicated to stamping it out, malware must be very good to survive for long.  Many or most strands will be identified and wiped out.  Those that survive will be scary indeed.

I don't know when we're going to get the first malware in the wild that can truly modify its own capabilities, rather than just its signature.  Maybe its already out there.  How complex is it getting?  At what point is it going to exceed its creators wildest expectations?  At what point will it begin exhibiting behaviors that will appear to demonstrate creativity and innovative problem solving?  A what point does it become self aware?

Whenever that happens, I don't know if we'll know what to do.  We're going to need help.

Maybe we can ask Watson.

The Limits of Reason

In the late eighteenth and early nineteenth centuries, a device known as the Mechanical Turk toured Europe and the United States, astounding it’s audiences by playing chess with an impressive degree of skill.  Many leading thinkers and engineers of the day were utterly convinced that it was a true chess playing machine, although there were persistent suspicions that perhaps the device (which included a large cabinet, apparently filled with gears and cogs) hid a real human chess player, possibly a dwarf, in its interior.

One of the most convincing exposes on the subject was written by Edgar Allen Poe in 1836.  His article was printed in the Southern Literary Messenger, and elicited responses from numerous newspapers and magazines including the New Yorker and the Baltimore Gazette.  It was well written, and completely correct in it’s conclusions – the Mechanical Turk was eventually revealed as a fraud.  However, it’s interesting to note that some of the most critical arguments employed by Poe were utterly incorrect.

His primary point was that as a mechanical device, it must necessarily be 100% fixed and determinate.  From this foundation, he makes several observations about the nature of the machine: “The moves of the Turk are not made at regular intervals of time, but accommodate themselves to the moves of the antagonist – although this point (of regularity), so important in all kinds of mechanical contrivance, might have been readily brought about by limiting the time allowed for the moves of the antagonist.”

He goes on to make a very interesting argument about the nature of the machine's purported algorithm: “The Automation does not invariably win the game.  Were the machine a pure machine, this would not be the case – it would always win.  The principle being discovered by which a machine can be made to play a game of chess, an extension of the same principle would enable it to win a game; a further extension would enable it to win all games – that is, to beat any possible game of an antagonist.  A little consideration will convince any one that the difficulty of making a machine beat all games is not in the least degree greater, as regards the principle of the operations necessary, than that of making it beat a single game.”

I’m not aware that any of Poe’s contemporaries had any disagreements with these points.  However, from our modern perspective, it’s easy to see that he was utterly incorrect.  We have chess playing machines today.  And what we know about the principles of making them play chess did not inevitably lead to an understanding of how to make them play perfect chess – it took decades of improvement (mostly on the hardware side) to evolve from the first primitive chess program to Deep Blue, which finally defeated world chess champion Gary Kasparov.  Even now that we've left human players far behind, nobody would claim that any computer in the foreseeable future could play a “perfect” game of chess (a feat which would require calculating every conceivable move on the chess board – a trivial exercise with a game like tic-tac-toe, but infeasible on a 64 square chess board).  Nor has our success at developing computer chess programs led to similar success teaching computers to master the much more fluid game of Go.

What would it have taken to convince Poe that he was wrong, especially when his ultimate conclusion was correct?  We could try to introduce him to the concept of heuristics, or allowing variable response times through the use of triggers, or even introducing the concept of indeterminacy to a system by seeding a complex algorithm with an external (and completely random) value.  Maybe this would have worked.  But none of this would be nearly as effective as simply having him grow up in a world where computers play chess all the time, and nobody thinks very much about it.

We like to think of ourselves as living in the age of reason and critical thinking.  We like to point at our technological progress and say “Here is where logic and reason have taken us.  They will solve any problem.”  We forget how much of what we know is built not on reason, but on trial and error, iterative improvement, and pure luck.