Beyond p(doom): How human agency is shaping real‑world AI outcomes

Artificial intelligence is increasingly discussed in extremes. At one end sits a utopia of human flourishing through technology; at the other stands p(doom) — the idea that AI could escalate into catastrophic outcomes. In contemporary AI discourse, p(doom) represents an individual’s assessment of the probability that advanced AI systems might cause existential harm. Originating within rationalist and AI safety communities, it has expanded into a common reference point across technical, academic and policy discussions. As large-scale AI models advance, timelines to artificial general intelligence shorten and expert opinions diverge, the visibility of p(doom) has intensified.

The challenge is that, while p(doom) can be a useful conversation starter, it is not a practical risk management tool. It collapses a wide range of scenarios into a single figure, often without a shared time frame, explicit assumptions or a clear connection to organizational decision-making. For organizations attempting to manage AI risk in practice, this level of abstraction is insufficient. At the same time, focusing exclusively on AI’s immediate impacts while overlooking risks emerging over the next three to 10 years may prove highly disruptive; strategic oversight of these near-horizon risks is precisely where it is most needed.

AI risk is not one thing

One reason AI remains difficult to govern is that it is often described as though it were a single category — “AI risk”. In reality, AI generates multiple distinct failure modes. The narratives embedded in p(doom) discussions typically fall into several thematic groups:

• Loss of control/misaligned AI. Goals diverging from human intent due to mis-specification, instrumental convergence or misalignment
• Rapid uncontrolled growth. Faster acceleration of capabilities than oversight, occasionally framed as“hard takeoff” or recursive self-improvement
• Harmful human use. Cyberattacks, disinformation, biological misuse or authoritarian surveillance
• AI‑enabled social collapse. Economic shocks, automated systemic failures or large-scale interdependent infrastructure disruptions
• Accidental technical or systemic failures. Unexpected behavior in high-stakes systems, control instabilities or cascading errors
• Multi‑agent failures. Emergent conflicts or coordination failures across interacting systems For most organizations, the more useful framing is not “What is our p(doom)?” but “Which AI-related failure modes already interact with our business model and how could they escalate under pressure?” Understanding consequence pathways, tolerance thresholds and decision impacts is more valuable than abstract labels.

From current AI failures to future losses

Recent incidents illustrate how AI-linked failures propagate through real systems. A fatal autonomous vehicle accident highlighted how design assumptions, governance gaps and weak human intervention can combine when systems are over-trusted. A deepfakeenabled fraud case demonstrated that AI could strengthen deception while operating legitimately within financial controls. Research in system operations shows how hallucinated or incorrect model outputs can create risks when operational teams rely on them too quickly. The narratives embedded into p(doom) discussions typically fall into several thematic groups exemplifying the diversity of this topic.

Multi-agent simulations show runaway dynamics that remain hidden until costs spike or systems diverge materially from expectations.

These events share a common pattern: Failures arise when adoption, automation and dependency outpace visibility, challenge and control. For risk and insurance professionals, this has direct implications. A model error can quickly shift categories — from technical flaw to conduct risk, customer outcome issue, regulatory problem or resilience challenge — depending on where it lands. A single AI-enabled impersonation can cascade into breakdowns in verification, authorization and financial controls. Bias in a model may start as a data quality problem but escalate into discrimination claims and regulatory scrutiny.

As with any transformation happening at speed, recent events call attention to the focus that needs to be given to where new technology has been introduced to the business from a risk management point of view. In this context, it is how specific AI failure could move through the business and where effective intervention is possible.

A practical framework for managing AI now

AI systems do not fail catastrophically by default. They fail when human agency is absent, weak or outpaced by adoption. Organizations that invest in technical controls, cultivate internal knowledge, recognize the pervasiveness of AI across their operations and operate within strong regulatory frameworks can harness AI as a transformational asset without incurring outsized systemic risk.

Leaders face a choice: Treat AI as an unpredictable force and react to failures afterward or shape AI outcomes through deliberate design, oversight and governance. The space between fiction and failure is defined not by model capability but by human agency. Strengthening that agency now will determine whether AI amplifies risk or becomes a resilient, responsible and strategically powerful capability.

What management should focus on

A recurrent feature of p(doom) discourse is unrestrained AI capability growth paired with diminishing human agency. In practice, four dimensions of human agency can anchor AI deployment in reality and away from speculative extremes:

Technical controls

Can the organization meaningfully constrain and override AI behavior in live environments? Misinterpretation, hallucination, drift and misuse become harmful when controls are absent or too weak. AI does not inherently understand legality, fairness or ethics; guardrails such as access controls, monitoring, evaluation and attack protections are essential.

• How are technical controls implemented and tested in your organization?
• Have they prevented any runaway processes or security breaches?
• How is accuracy, robustness and reliability validated?

Regulation

Regulation matters because AI raises questions of accountability, documentation, fairness and customer impact. Laws set minimum requirements; governance, culture and ethics determine how far organizations go beyond compliance. Trustworthy AI is increasingly seen as a competitive advantage.

• What internal standards go beyond regulatory minimums?
• How are regulatory developments shaping deployment decisions?
• Can the organization produce documentation for regulators or clients?

Knowledge

Capability gaps create control weaknesses. If professionals become dependent on AI without maintaining core judgment skills, resilience weakens. In addition to this critical responsibility to maintain core knowledge outside of AI, there is a growing requirement for a depth of technical knowledge of how the AI is being used to support and complement these core skills. Technical knowledge must be distributed across the organization — not confined to data science teams — especially because employee decisions drive model use, escalation and oversight.

• How is knowledge maintained and challenged outside of AI use?
• How often is the organization upskilled on AI risks and use?
• How do you ensure that AI knowledge is complementing work, not completing work?

Adoption strategy

Unseen use equals unmanaged exposure. Visibility is the foundation of resilience. Widespread AI adoption creates systemwide dependencies. Concentration of AI infrastructure raises correlated risk: if many insurers use similar models or rely on the same cloud resources, failures can scale rapidly.

• Do you have visibility to what models run beneath your AI?
• Which critical workflows depend on AI and what are the contingencies?
• How are new AI use cases monitored?

What boards require

Boards should ensure that operational AI insights translate into governance. Their role is to challenge whether the organization has moved beyond labels and into consequence, tolerance, ownership and action.

Boards should ask:

• Which strategic assumptions would fail under plausible AI scenarios?
• Where is the organization over-reliant on a single AI-dependent future?
• Which AI-related risk combinations could push the organization beyond risk tolerances?
• How does capital allocation today affect future AI resilience and optionality?
• Who is accountable for turning AI insight into an actual decision?

Closing the space between fiction and failure

Discussions around p(doom) and existential AI risk can give the impression that future risks will emerge as dramatic, high-profile events that capture attention and disrupt the industry in a single moment. In practice, however, the impact is far more likely to accumulate through everyday decisions and processes. Losses tend to arise incrementally: a model that is over-trusted, a workflow scaled beyond its effective controls, a fraud safeguard quietly bypassed, a pricing assumption embedded too broadly or a team that gradually loses the confidence to operate without the system it is meant to oversee. These are subtle shifts, but over time they can compound into material risk.

While the naturally interesting and theoretically challenging p(doom) discussions can open the door to conversations about AI risk, turning that abstraction into something practical and actionable is a necessary next step.The task centres on maintaining control over where and how AI is deployed and embedding that control into disciplined, repeatable governance. In this context, the most valuable handoff is a clearly defined issue: an identifiable failure mode, a well understood consequence pathway, an agreed action plan and a named owner who is accountable for delivery.

That is how AI risk becomes governable.