Appendix N: Bucknall Case Study - Near-Term AI as Existential Risk Factor

N.1 Introduction to the Bucknall Extraction

The manual extraction of Bucknall and Dori-Hacohen’s “Current and Near-Term AI as a Potential Existential Risk Factor” Bucknall and Dori-Hacohen (2022) serves as more than a validation exercise—it provides a window into how experts navigate the extraction process and where human judgment proves essential.

This paper makes an ideal test case for several reasons. First, it explicitly discusses multiple causal pathways from near-term AI to existential risk, providing rich material for extraction. Second, it bridges technical and policy concerns, testing the system’s ability to handle interdisciplinary arguments. Third, its relatively recent publication (2022) means it engages with current AI capabilities rather than speculative future systems.

The manual extraction process, conducted independently by Johannes Meyer and Jelena Meyer, revealed both the possibilities and limitations of formalizing complex sociotechnical arguments.

N.2 The Extracted Structure

The manual extraction revealed a hierarchical argument with existential risk as the root outcome, influenced by multiple pathways:

[Existential_Risk]: Increase in existential risks for humanity. {"instantiations": ["TRUE", "FALSE"]}

- [Unaligned_AGI_Risk]: Unaligned artificial general intelligence causes existential risk. {"instantiations": ["TRUE", "FALSE"]}
  - [State-State_Relations]
- [Near_term_AI]: Even if not unaligned AGI, near term AI can act as intermediate risk factor. {"instantiations": ["TRUE", "FALSE"]}
  - [State-State_Relations]: AI arms race dynamic inhibits international coordination, diverting resources from other pressing issues {"instantiations": ["TRUE", "FALSE"]}
    - [Cybersecurity]: Probably enhances Cyber-Attack-Offense, may intensify cyber warfare. {"instantiations": ["TRUE", "FALSE"]}
  - [State-Corporation_Relations]: Corporations have a lot of power and might have misaligned goals with society {"instantiations": ["TRUE", "FALSE"]}
  - [Stable_Repressive_Regime]: More repressive instruments, possibility of stable repressive regime. {"instantiations": ["TRUE", "FALSE"]}
    - [State-Citizen_Relations]: AI helps regime monitor citizens {"instantiations": ["TRUE", "FALSE"]}
  - [Compromised_Political_Decision_Making]: AI can compromise political decision making. {"instantiations": ["TRUE", "FALSE"]}
    - [Social_media_and_Recommender_Systems]: Influence of AI in social media on public opinion. {"instantiations": ["TRUE", "FALSE"]}
- [Nuclear]: Probability that nuclear conflict escalates to end civilisation. {"instantiations": ["TRUE", "FALSE"]}
  - [Compromised_Political_Decision_Making]
- [Biological]: Probability that a natural or engineered pandemic poses existential risks. {"instantiations": ["TRUE", "FALSE"]}
  - [Compromised_Political_Decision_Making]
  - [Social_media_and_Recommender_Systems]
- [Natural]: Non-human caused existential risks, seem unrelated with AI. {"instantiations": ["TRUE", "FALSE"]}
- [Environmental]: Probability of climate catastrophe. {"instantiations": ["TRUE", "FALSE"]}
  - [Compromised_Political_Decision_Making]
  - [AI_resource_consumption]: Current AI models consume large amounts of energy having environmental impacts. {"instantiations": ["TRUE", "FALSE"]}
  - [Social_media_and_Recommender_Systems]
[Existential_Risk]: Increase in existential risks for humanity. {"instantiations": [TRUE", "FALSE"]}

- [Unaligned_AGI_Risk]: Unaligned artificial general intelligence causes existential risk. {"instantiations": [TRUE", “FALSE”]}
  - [State-State_Relations]
- [Near_term_AI]: Even if not unaligned AGI, near term AI can act as intermediate risk factor. {"instantiations": [TRUE", "FALSE"]}
  - [State-State_Relations]: AI arms race dynamic inhibits international coordination, diverting resources from other pressing issues {"instantiations": [TRUE", “FALSE”]}
    - [Cybersecurity]: Probably enhances Cyber-Attack-Offense, may intensify cyber warfare. {"instantiations": [TRUE", “FALSE”]}
  - [State-Cooperation_Relations]: Cooperations have a lot of power and might have misaligned goals with society {"instantiations": [TRUE", “FALSE"]}
  - [Stable_Repressive_Regime]: More repressive instruments, possibility of stable repressive regime. {"instantiations": [“TRUE", "FALSE"]}
    - [State-Citizen_Relations]: AI helps regime monitor citizens {"instantiations": [TRUE", "FALSE"]}
  - [Compromised_Political_Decision_Making]: AI can compromise political decision making. {"instantiations": [“TRUE", "FALSE"]}
    - [Social_media_and_Recommender_Systems]: Influence of AI in social media on public opinion. {"instantiations": [TRUE", "FALSE"]}
- [Nuclear]: Probability that nuclear conflict escalates to end civilisation. {"instantiations": [TRUE", "FALSE"]}
  - [Compromised_Political_Decision_Making]
- [Biological]: Probability that a natural or engineered pandemic poses existential risks. {"instantiations": ["TRUE", “FALSE"]}
  - [Compromised_Political_Decision_Making]
  - [Social_media_and_Recommender_Systems]
- [Natural]: Non-human caused existential risks, seem unrelated with AI. {"instantiations": ["TRUE", “FALSE"]}
- [Environmental]: Probability of climate catastrophe. {"instantiations": ["TRUE", “FALSE"]}
  - [Compromised_Political_Decision_Making]
  - [AI_resource_consumption]: Current AI models consume large amounts of energy having environmental impacts. {"instantiations": ["TRUE", “FALSE"]}
  - [Social_media_and_Recommender_Systems]

N.3 Key Insights from Manual Extraction

The extraction process revealed several important patterns:

Multi-Path Risk Architecture: Unlike simpler models with linear causal chains, Bucknall’s argument presents existential risk as emerging from multiple, interacting pathways. Near-term AI doesn’t directly cause existential catastrophe but acts as a risk multiplier across various domains.

Institutional Intermediaries: The model highlights how AI affects existential risk through institutional mechanisms—state relations, corporate power, political decision-making. This sociological sophistication challenges purely technical risk models.

Reused Components: Several nodes (like “Compromised_Political_Decision_Making”) influence multiple risk pathways. This reuse reflects how certain AI impacts create systemic vulnerabilities rather than isolated risks.

Implicit Probabilities: While the structure emerged clearly, the paper provides few explicit probability estimates. Extractors had to infer likely ranges from qualitative language like “probably enhances” and “may intensify.”

N.4 Extraction Challenges and Decisions

The manual extraction process faced several decision points that illuminate the broader challenge:

Granularity Choices: How finely should arguments be decomposed? The extractors chose to keep “State-State_Relations” as a single node rather than breaking it into specific mechanisms, judging that further decomposition would lose coherence.

Temporal Ambiguity: The paper discusses both current impacts and future possibilities without always clearly distinguishing. Extractors had to decide whether to model these as separate nodes or probability variations.

Causal vs. Definitional: Some relationships blur causation and definition. Is “AI helps regime monitor citizens” a cause of “Stable_Repressive_Regime” or part of what defines it? Such philosophical questions arose repeatedly.

Background Assumptions: The paper assumes familiarity with existential risk frameworks. Extractors had to decide how much implicit framework to make explicit in the model.

N.5 Validation Insights

Comparing the independent extractions revealed:

Structural Convergence: Both extractors identified the same major causal pathways and risk factors, suggesting the paper’s argument structure is reasonably unambiguous.

Naming Variations: Different choices in node naming (“State-Corporation_Relations” vs. “Corporate_Power”) created surface disagreements while representing the same concepts.

Probability Divergence: Where extractors attempted to quantify relationships, estimates varied by 20-30 percentage points, confirming that probability extraction faces inherent subjectivity.

Consistent Struggle Points: Both extractors found certain passages challenging, particularly where the paper discusses complex feedback loops between AI development and geopolitical dynamics.

N.6 Implications for Automated Extraction

This manual extraction exercise validates several AMTAIR design decisions:

  1. Two-Stage Architecture: The clear separation between structural and probability extraction in human practice supports the system’s two-stage approach.

  2. Human-in-the-Loop: Even expert human extractors made judgment calls requiring domain knowledge, confirming that full automation remains premature for high-stakes applications.

  3. Variation Acceptance: The legitimate differences between expert extractions suggest systems should support multiple interpretations rather than seeking single “correct” outputs.

  4. Metadata Importance: Preserving descriptions and context proves essential for interpreting formalized models, validating the BayesDown format’s hybrid approach.

N.7 From Manual to Automated

How would AMTAIR handle this same extraction task? Based on similar complexity arguments:

  • Structure Extraction: The system would likely capture the main risk pathways successfully, though it might struggle with the nuanced institutional relationships.

  • Node Identification: Explicit risk factors would extract cleanly, but implicit mechanisms might be missed or mischaracterized.

  • Probability Assignment: Without explicit estimates in the text, the system would rely on linguistic mappings, likely producing reasonable but imprecise values.

  • Processing Time: What took human experts 4-6 hours would complete in under 2 minutes, enabling analysis at scale despite lower per-document precision.

N.8 The Value Proposition Crystallized

The Bucknall case study crystallizes AMTAIR’s value proposition. Perfect extraction remains elusive even for human experts. But rapid, good-enough extraction that captures primary arguments and enables systematic comparison across many documents provides genuine value. The goal isn’t replacing human judgment but augmenting human capacity to synthesize complex arguments about existential risk.

In a world where relevant arguments proliferate faster than experts can read them, tools that extract 80% of the structure 100 times faster than humans transform from nice-to-have to necessary-for-coordination. The Bucknall extraction shows both why this transformation matters and how far we still have to go.