Microsoft CEO: Cutting-edge models should be regarded as internal risks, and control must be placed outside the models

Microsoft CEO Satya Nadella published an article discussing the trust framework for the era of superintelligence. He stated that traditional software can trace behaviors back to specific code paths, whereas current superintelligent systems lack this mechanistic understanding and cannot attribute model behaviors and outputs to specific training data or weight configurations. However, they are being deployed and are accessing sensitive data, executing critical actions on behalf of others. He emphasized that responsibility for intelligent behavior cannot be outsourced to model providers, nor can superintelligence be accepted or rejected as a series of nested black boxes.

Nadella proposed that the supply of intelligence should be separated from the authorization of intelligence, surrounding nondeterministic models with deterministic system design, human control, and operational procedures, and establishing industry standards when existing standards are insufficient. He suggested managing cutting-edge models with closed-source and open-source weights as internal risks, not because they are inherently malicious, but because agents with sufficient capability that can access important systems may make mistakes or be compromised. Relevant practices include establishing identities, restricting permissions, recording activities, and setting isolation boundaries, with the transparency of the model's reasoning chain becoming a non-negotiable starting point.

He further stated that permissions and action controls must be placed outside the model, and the model, orchestration layer, and action space should be separated. Principles include avoiding reliance on a single model as the sole dependency for important outcomes or for verifying its own work, ensuring that critical actions leave tamper-proof and human-readable evidence, continuously testing for failures, attacks, and boundary conditions, allowing organizations to independently decide on access and action permissions, validating independently of the intelligence being verified, presuming that the model has been compromised and supporting mid-course pauses or shutdowns, and timely disclosing and sharing lessons within the industry when failures or compromises occur. He concluded that the most trustworthy superintelligent systems are those that require the least trust in the model itself.