Nvidia has just launched Open Agent Safety Platform with more than 100 industry partners, including Anthropic, Hugging Face, IBM, Microsoft and Palantir.
Nvidia CEO Jensen Huang has called the platform "the beginning of an open ecosystem to build the trust layer for safe agent systems. Together, we are building the foundation of the AI economy."
Announcement
Core principles for building an agent system:
- Policy needs to be verifiable: Before an agent runs, a prover shows that its policy cannot escape the intent of the operator.
- Enforcement must be out of band: The controls do not live inside, or within reach of the agent. The agent does not need to know it is being watched.
- The path to the model (the brain) is the control point: An agent cannot act without its next thought. By controlling the path to the model, you own both the best observation point and also the kill switch to interrupt it if you need to.
- Scale agent authority with the ability to inspect its thinking: The more an agent can do, the more its reasoning needs to be visible. An advantage of open models is that the entire reasoning space and activations are all visible.
- Applying the shared responsibility model: Labs, enterprises, and hardware providers each own a layer, just like the cloud today. The agent runtime and its policy language need to be open so any provider can plug in.
Three open agent safety platform layers:
- The application: What end-users are building. Contains the necessary primitives for mission success: models, harnesses, tools, data, and support scripts and programs.
- The runtime: Projects the application layer onto infrastructure. Orchestrates the agentic workload onto infrastructure that meets end-user requirements (end-user workstation, edge device, data center) and provides continuous monitoring and real-time policy enforcement and governance.
- The infrastructure: Concrete hardware resources used to execute agentic workloads. Network calls to upstream services, databases and filesystem access, general purpose compute for tool and code execution, accelerated compute for safety monitoring and increased workload density.
------------------------------
Todor Kostov
CFO Consultant and Board Advisor
------------------------------