Robots, genetics pose grave danger: scientist

Globe and Mail Online

Technology close to being out of control,
respected Silicon Valley figure writes


Tuesday, March 14, 2000

San Francisco -- The co-founder of one of Silicon Valley's top technology companies believes scientific advances may be ushering humanity into a nightmare world, where supersmart machines force mankind into extinction.

In an appeal published in the April issue of Wired magazine, Sun Microsystems' chief scientist, Bill Joy, urges technologists to reconsider the ethics of the drive toward constant scientific innovation.

"We are being propelled into this new century with no plan, no control, no brakes," Mr. Joy writes.

"The last chance to assert control -- the fail-safe point -- is rapidly approaching."

His article comes as a rare cry of caution in an industry that thrives on relentless and often unplanned advances and is now riding the boom of a "new economy" expansion attributed to technological progress.

The warning is all the more disturbing because of the author's credentials. A leading computer researcher who developed an early version of the Unix operating system, Mr. Joy has more recently pioneered the development of software such as Java and was co-chairman of a U.S. presidential commission on the future of information technology.

His fears focus on three areas of rapid change.

The first, robotics, involves the development of "thinking" computers that within a matter of three decades could be as much as a million times more powerful than those now available. Mr. Joy sees this as laying the groundwork for a "robot species" of intelligent robots that can create evolved copies of themselves.

The second, genetics, deals with scientific breakthroughs in manipulating the very structure of biological life. Mr. Joy noted that while this has led to benefits such as pest-resistant crops, it also has set the stage for new man-made plagues that could literally wipe out the natural world.

The third, nanotechnology, involves the creation of objects on an atom-by-atom basis, which before long could be harnessed to create smart machines that are microscopically small.

All three of these technologies share one characteristic absent in earlier dangerous human inventions such as the atomic bomb: They could easily replicate themselves, creating a cascade effect that could sweep through the physical world in much the same way a computer virus spreads through the cyberworld.

"It is no exaggeration to say we are on the cusp of the further perfection of extreme evil," Mr. Joy writes.

"An evil whose possibility spreads well beyond that which weapons of mass-destruction bequeathed to nation states, on to surprising and terrible empowerment of extreme individuals."

Executives attending PC Forum, a technology-industry conference in Scottsdale, Ariz., had mixed reactions to Mr. Joy's grim view.

"You can't argue with the incredible improvements in thought, civilization, productivity," said Michael Campbell of SAP America Inc., which develops software to help corporations manage their business. "The world is changing, and I think people would say dramatically for the better."

Steve Kirsch, founder of Infoseek and now chief executive of a new start-up called Propel, said he sees many instances where science and research are not moving fast enough, such as in the area of embryonic stem-cell research. "If anything . . . we are moving the opposite way, not faster," he said.

But others said Mr. Joy's fears of technology running out of control are not entirely off the mark.

"From the sense that technologists fail to fully study or bother to understand the long-range implications of technology, Bill Joy is right on the mark," said Steve Larsen, senior vice-president of marketing at Net Perceptions, which develops personalization software for the Internet.

Mr. Joy said his dark vision of the potential threat to humanity posed by technology has led him to reconsider his own contributions to the field.

"I have always believed that making software more reliable, given its many uses, will make the world a safer place," he writes. "If I were to come to believe the opposite, then I would be morally obligated to stop this work. I can now imagine such a day may come."

Mr. Joy does hold out some hope, saying humanity's effort to control the threat of nuclear and biological weapons was evidence of the strength of the species' instinct for self-preservation.

Category: 4. Ethical and Social Concerns Arising out of Biotechnology