NVIDIA and Lakera AI Propose Unified Framework for Agent Safety

NVIDIA and Lakera AI have unveiled a unified security and safety framework to address the challenges posed by autonomous AI agents in enterprise environments.

This collaboration advances the security of agentic systems—AI systems capable of independent planning, tool use, and multi-step task execution—making them safer for deployment.

The research, conducted by scientists from both organizations, views safety and security as properties emerging from interactions among various components in agentic systems rather than as inherent attributes of individual models.

Agentic systems differ from traditional language models as they can invoke tools, access external data, make autonomous decisions, and interact with users across multiple steps, creating new vulnerabilities that traditional evaluation approaches may miss.

Agentic systems introduce unique risks through their compositional architecture, unlike isolated large language models (LLMs) that have undergone extensive evaluations. These systems may involve multiple sub-agents that can invoke tools and APIs, consult RAG, and interact with an external environment.

The framework identifies unique agentic risks, including tool misuse, cascading action chains, unintended control amplification, and multi-agent interactions that cannot be adequately assessed using conventional security metrics alone. The researchers propose examining safety and security through a unified lens to prevent user harm.

The framework features an operational risk taxonomy that combines traditional safety and security concerns with uniquely agentic risks, prioritized by impact and exploitability. At its core is a dynamic assessment methodology using specialized AI agents for risk discovery, evaluation, and mitigation. The framework employs continuous, context-aware evaluation through sandboxed AI-driven red teaming, rather than relying solely on static pre-release testing.

Researchers introduced Agent Red Teaming via Probes (ARP), a methodology allowing targeted security testing throughout an agentic workflow. This approach provides developers with insights into how threats propagate through system components, independently of upstream changes, offering granular visibility into security weaknesses.

The framework's effectiveness was demonstrated through an extensive case study of NVIDIA's AI-Q Research Assistant (AIRA), a tool synthesizing information from enterprise databases and web search. The study included over 10,000 attack and defense executions across 22 threat scenarios covering nine risk categories.

The research revealed differential attack propagation patterns, with direct user input attacks amplifying through the processing pipeline, while attacks from external data sources were progressively attenuated. This insight informs targeted defense strategies.

The framework achieved approximately 50% risk reduction through targeted guardrails and demonstrated continuous tracking of security improvements across agent versions.

NVIDIA and Lakera AI have released the Nemotron-AIQ Agentic Safety Dataset, containing over 10,000 trace files from their experiments, to enable the research community to advance agentic safety evaluation. The framework’s focus on contextual, layered defenses represents a practical approach to balancing robust security with acceptable system performance.

This collaborative framework marks a significant milestone in establishing methodologies for securing autonomous AI systems, addressing a critical gap in enterprise AI deployment guidance.

Based on reporting by GBHackers.