Neel Somani on What Every CEO Should Understand About AI Risk

AI Governance: Managing Systems Beyond Human Understanding

Executives in large corporations face a critical question: How do we govern systems we do not fully understand? This is particularly relevant for AI systems, where traditional risk management strategies prove inadequate. These strategies typically rely on predictability, code auditing, and compliance certification, which do not apply to AI systems that operate through mechanisms even their creators struggle to explain.

AI Risk: Critical Questions for Leaders

Organizations must address key questions regarding AI governance:

Can the cause of a system failure be identified? Can the issue be resolved without introducing new problems? Can the effectiveness of the fix be confirmed?

The repercussions of these questions extend beyond reputational damage or regulatory fines, touching on fundamental organizational control.

Modern AI systems often function as black boxes, providing "post hoc" explanations for decisions. However, these explanations may not accurately reflect the system's underlying logic. This gap between explanation and mechanism creates two significant risks: hidden failure and intervention fragility.

Hidden failure occurs when a system's unclear logic leads to undetected issues, causing failures that standard testing cannot prevent. An intended improvement can inadvertently impair other performance areas.

Intervention fragility arises when resolving an identified issue introduces new, harder-to-detect problems. This is a pressing operational concern for organizations relying on AI for significant decisions.

Neel Somani emphasizes the importance of debuggability , which involves localizing failures, intervening predictably, and verifying that interventions maintain desired behaviors. Unlike interpretability, debuggability focuses on understanding and controlling the internal mechanisms driving specific behaviors.

Localization: Identify specific mechanisms responsible for a behavior and distinguish between causative and correlative mechanisms. Intervention: Modify mechanisms predictably to address undesirable behaviors without introducing new issues. Certification: Make falsifiable claims about model behavior within defined domains, offering guarantees beyond probabilistic assurances.

Traditional governance relies on transparency and accountability, which are challenged by AI's black-box nature. Emerging regulations like the European Union's AI Act and standards from organizations such as NIST emphasize explainability and accountability. However, compliance alone does not ensure operational capability to fix systems when they fail.

Debuggability shifts the focus from compliance to operational capability, ensuring that the root causes of failures are identified, corrected confidently, and verified effectively.

Somani compares AI systems to safety-critical software systems. While global guarantees are unattainable, domain-specific assurances can be made, such as proving that certain pathways cannot bypass safeguards.

Achieving debuggability requires investment in formal verification techniques, which use mathematical proofs to establish software properties. These techniques, traditionally applied to safety-critical systems, are being adapted for AI.

Organizations must decide whether to wait for these methods to mature or to invest in building capabilities now. Investment involves training teams in AI and formal methods, establishing standards for debuggability, and choosing vendors prioritizing verifiable systems.

Discussions about AI risk often focus on external threats, but the primary risk is the possibility of AI failure during normal operations. Organizations must build systems capable of rigorous debugging to ensure effective governance and control.

For CEOs, the question is not about AI's impact on the industry but whether their organization has the capacity to govern AI effectively when necessary.

Based on reporting by TechBullion.