Skip to main content

Security and Non-Deterministic I/O

Traditional access-control models assume the operator can predict, at policy-design time, which resources a workload will ask for. ACLs name principals and resources; RBAC names roles and permissions. The assumption holds when application code is written by people who know what it needs to touch.

The assumption breaks when an LLM is in the loop. The model's tool calls, query patterns, and downstream actions are functions of a runtime input (a prompt) that may have been written by an attacker. The operator can no longer predict which resources the workload will ask for, because the access pattern is a non-deterministic function of an untrusted input. This is the structural problem that AI agent security has to solve, and it is why prompt filtering alone is insufficient.

This is where capability-based security becomes essential. Even when an attacker fully compromises the model's decisions, the workload can only reach capabilities the operator granted it through the manifest. That bound is static, inspectable, and unaffected by what the model decides.

Key takeaways
  • Access control models that assume a fixed, designer-known access pattern break when an LLM generates tool calls at runtime.
  • Prompt injection is a structural problem: specifically, the Confused Deputy problem applied to LLM tool-calling.
  • Capability-based security gives you the only defense that holds when the model is fully compromised: a manifest-set reach that the model literally cannot widen.

Why traditional models break​

When application code is written by humans, permission scope is bounded by what the code requires at design time.

When the same service has an LLM inside it generating tool calls, the access pattern is no longer derivable from the code. It is a function of:

  • The model's training distribution, which is non-deterministic given any one prompt.
  • The user's prompt, which is often untrusted.
  • Tool descriptions and intermediate outputs that feed back into the model's context window.

This is the structural reason prompt injection (Willison, 2022) and indirect prompt injection (Greshake et al., 2023) are not bugs that can be patched away. They are consequences of mixing instructions and data in a single context window. The OWASP Top 10 for Large Language Model Applications lists prompt injection as its number-one risk for exactly this reason. Filtering prompts and sanitizing outputs raises the bar — and is a meaningful defense layer in any LLM security or AI agent security program — but it does not prove a bound. The defense has to live somewhere the attacker can't reach.

Why static bounds matter​

Capability-based security gives you a bound that lives outside the model's reach.

A WebAssembly component holds only the capabilities its manifest grants. The model's output can drive which of those capabilities the component invokes and with what arguments, but it cannot grow the set. If the model says "exfiltrate to attacker.com" and attacker.com is not in the allowedHosts allowlist, the request fails at the host plugin layer.

This is the same problem the Confused Deputy (Hardy, 1988) describes, applied to LLM tool-calling: a trusted intermediary (here, the agent) is tricked by an untrusted caller (the prompt) into using its authority on the caller's behalf. Capability-based systems prevent the attack structurally, and that is what makes them suitable for agentic AI security in a way that filtering, monitoring, and runtime guardrails are not.

Two-panel diagram comparing AI workloads under ambient authority versus capability bounds. In both panels, the same chain runs top to bottom: attacker prompt feeds into an LLM, which feeds into the workload. Left panel "Ambient authority + LLM": the workload is a process with ambient authority, and solid arrows reach three targets (api.anthropic.com, attacker.com, filesystem). Right panel "Capability bounds + LLM": the workload is a Wasm component with manifest-granted capabilities, a single solid arrow reaches the granted target (api.anthropic.com), and dashed lines mark attacker.com and filesystem as unreachable. The structural defense is the manifest, which the model cannot widen.

This is the same invariant that made the Principle of Least Authority useful in 1975, applied to a new failure mode. The set of things a workload can ultimately cause to happen is fixed when the manifest is written, not when the model is called. Even a fully compromised model is bounded by the capabilities the platform team granted at deploy time.

Bounds in Cosmonic Control​

A typical AI workload in Cosmonic Control might be an agent or MCP server running as a Wasm component inside a WorkloadDeployment. The manifest declares everything the workload may reach:

apiVersion: runtime.wasmcloud.dev/v1alpha1
kind: WorkloadDeployment
metadata:
  name: customer-support-agent
spec:
  replicas: 3
  template:
    spec:
      hostSelector:
        hostgroup: default
      components:
        - name: agent
          image: ghcr.io/example/support-agent:0.3.0
          localResources:
            environment:
              config:
                LOG_LEVEL: info
            allowedHosts:
              - https://api.anthropic.com
              - https://kb.support.internal:8443
      hostInterfaces:
        - namespace: wasi
          package: keyvalue
          version: 0.2.0-draft
          interfaces:
            - store
          config:
            bucket: support-conversations

The agent component can:

  • Call the Anthropic API (api.anthropic.com), needed for the model.
  • Call the internal knowledge base (kb.support.internal:8443).
  • Read and write conversation state in the support-conversations keyvalue bucket.
  • Read its LOG_LEVEL config value.

Whatever a customer's prompt convinces the model to ask for, the agent cannot:

  • Make a request to attacker.com, an internal admin API, or any host not in allowedHosts.
  • Read another tenant's conversations or any other keyvalue bucket.
  • Read environment variables outside its declared config.
  • Touch the filesystem, spawn a process, or open arbitrary TCP connections.

The attack surface is defined in the manifest: a reviewer reading the YAML can see the upper bound of what this agent can do without trusting the component image, the model, or any chain of prompts that reaches the workload at runtime. This is what we mean by AI workload security: a bound that is set when the workload is deployed, and that the model cannot widen at runtime.

See Sandbox AI for the applied tutorial on MCP security: deploying an MCP server as a capability-bound Wasm component on Cosmonic Control.

Frequently asked questions​

What is AI agent security?

AI agent security is the discipline of bounding what an LLM-driven workload can actually reach at runtime (e.g., the hosts it can call, the data it can read, and the actions it can take) given that the access pattern is generated by the model, not specified by the developer. Defense layers include prompt filtering, output sanitization, sandboxing, and capability-based isolation; capability bounds are the only layer the model cannot widen by itself.

How does capability-based security apply to AI agents?

A WebAssembly component running an agent or MCP server holds only the capabilities its manifest grants. The model's output decides which of those capabilities to invoke and with what arguments, but it cannot grow the set. If the model is fully compromised, the maximum reach of the workload is still bounded by the manifest the platform team controls.

What is the difference between prompt injection and indirect prompt injection?

Prompt injection is when an attacker writes instructions directly into a prompt that an LLM-driven application processes. Indirect prompt injection is when an attacker plants instructions in data the model retrieves at runtime such as a web page the agent reads, a document it summarizes, or a tool output it consumes; the model then treats that data as instructions. Both are consequences of mixing instructions and data in a single context window, and both are listed by OWASP under LLM01 in the Top 10 for LLM applications.

How do you secure an MCP server?

An MCP server is a tool surface exposed to one or more LLM hosts; security depends on bounding what the server itself can reach. Deploying the MCP server as a WebAssembly component on a runtime that enforces capability-based security means the server can only call the specific hosts, read the specific configuration values, and access the specific data buckets declared in its workload manifest, regardless of what the connecting model decides to ask for.

What is the OWASP LLM Top 10?

The OWASP Top 10 for Large Language Model Applications is a community-maintained list of the highest-priority security risks for LLM-integrated applications. Prompt injection (LLM01) is its number-one entry. It is the closest thing to a canonical threat model for AI agent security and is the reference most other practitioner guides cite.

Why does prompt filtering not solve the problem?

Prompt filtering raises the bar but does not prove a bound. Filtering is a probabilistic defense: given a sophisticated enough attacker, prompt injection cannot be reliably distinguished from legitimate input, because both arrive in the same context window. A capability bound means that the workload cannot reach a resource that wasn't granted, no matter what the model is convinced to try. Filtering is useful as a defense layer, while capability bounds are useful as a defense floor.

Last updated on by Eric Gregory