As often happens, philosopher David Wallace gives the clearest statement of a point (from the earlier thread):

Let me have a go at the “normie” case for worrying about catastrophic if perhaps not quite existential risks, avoiding science fiction as much as possible. (The best science fiction is written by smart people who try hard to predict the future, so I don’t think “this is science fiction” is really a good objection, but I get why people are put off by it.)

Start with three observations. First: the trajectory of AI over the last thirty years, and more so over the last ten years, and even more so over the last five, is that AI has learned how to do more and more cognitive tasks that people had thought would be impossible or take decades. Chess; Go; protein folding; machine translation; natural-language communication; image recognition; programming; hacking; aspects of higher mathematics. And when it achieves human-level competence in these areas, it very often goes past them to superhuman-level competence. We have no systematic theory of AI and progress could stop tomorrow, but people have been predicting that for years and meanwhile progress has continued apace.

Second observation: increasingly many AI systems are agentic: that is, they’re agents in the world that act in goal-oriented ways, they’re not (all) just chatbots. Most of those agents are virtual, but a drone that killed three people in Ukraine last month appears to have been AI-controlled. As I and others upthread have said, whether this is ‘real’ intentionality isn’t the point: they behave like things with real intentionality.

Third observation: we have no reliable way to get AI agents to do what we tell them to, or to not do things we forbid (no reliable way to ‘align’ them, in the jargon of the AI community). That’s a consequence of the very opaque way AI systems are constructed: we don’t really know how they work, at the emergent whole-system level; they are grown and trained, not designed. It is also supported empirically by the last few months’ control-failure stories. The degree to which AI companies were carelessly culpable in some of those stories isn’t the point: these are *at least* the sorts of agents where, if you make a mistake, you will lose control of them and they will do advanced cognitive tasks you didn’t intend them to do.

So: we are building a large number of agents that can perform increasingly many cognitive tasks and can do some of them at superhuman levels on at least some axes; and our control of them is highly imperfect. Put that way, I think it’s *obviously* dangerous, but we can look at some specific risks (again, being as non-science-fictional as possible).

– AI is already better at hacking than humans on many axes: the HuggingFace attacks are the kind of thing that, this time last year, required nation-state-level resources. In the quite near term it is easy to imagine losing control of our computer networks, or large parts of them, to rogue agent swarms. That’s not an existential threat, in itself, but it could do catastrophic economic damage, get a lot of people killed directly or indirectly, and be dangerously destabilizing internationally.

– The Ukraine war makes clear that the future of war is drones. And the future of drones is probably autonomous – that is, AI – control. On pretty short timescales, especially given how the war is acting as an accelerant, it’s plausible that militaries will consist largely of networked, AI-controlled drone swarms. There are then obvious risks from control failure – and, indeed, other obvious risks if the swarms are aligned, but to bad actors. It is tempting to say “ok, don’t build autonomous drone swarms”, but the military advantages of building them will be so large that countries will resist unilateral restraint.

– Designing viruses with long onset times, high contagion, and high lethality is possible now but requires very large resources and is highly visible to intelligence agencies. There are lots of ways in which near-future AI could make this simpler, cheaper, and less detectable. If it became possible for terrorists to design and synthesize lethal bioweapons via AI and online DNA-ordering resources, it would lead to civilizational disaster. (This has been discussed a fair amount recently and it’s fair to say that experts are divided on how feasible it is – but to non-experts, “some experts say this is a serious risk, some say it isn’t, and the reasons they disagree aren’t legible to non-experts” averages out to a medium-size risk.)

– Even without AI, the current geopolitical moment is the most dangerous since at least the early 1980s. There are a number of ways in which AI could destabilize nuclear deterrence. One is that rogue or bad-actor-controlled AIs could confuse or spoof various of the early-warning systems that detect nuclear first use. Another is that deterrence relies to a large degree on the fact that ballistic missile submarines are undetectable; AI might threaten that on several axes (tracking by drone swarms, for instance) and make a nuclear first strike an attractive option in the event of great power war.

None of those risks is literally existential, but they are all catastrophic at the least and civilization-destroying at the worst. I think there are literally existential risks too – there are worse nightmares out there – but they involve rather more speculation, and from a practical point of view I’m not sure it matters much whether we are worried about AI literally killing everyone, or just about AI triggering a global catastrophe that kills billions and shatters industrial civilization.

It would be helpful to hear specific responses to this argument for being worried about AI. Please make sure to respond to the actual points Professor Wallace has made.