Your AI Agent Doesn't Need kubectl. It Needs a Bouncer.

2025-11-10

Raw kubectl access for an AI agent is a cluster incident waiting to happen. Build a bouncer: a secure layer that lets agents act without running wild.

Every week another team shows me their new AI agent. It parses alerts, writes runbooks, even drafts incident post-mortems. Then they demo it live — and I watch the agent call kubectl delete pod on a production namespace.

The room goes quiet.

This is the pattern I keep seeing: teams build impressive AI capabilities and wire them directly to kubectl or the Kubernetes API server. No guardrails. No audit trail. No limits. The agent is brilliant — until it isn't, and then it has cluster-admin.

The fix isn't to take away the agent's access. It's to give it a bouncer.

AI agent architecture without guardrails — direct kubectl access to the cluster

The Problem with Direct kubectl Access

A bouncer at a club doesn't care how charming you are. They check your ID, verify you're on the list, and turn you away if the venue is at capacity — no matter what you say. Your AI agent needs the same treatment.

When agents call kubectl directly, you're handing them a master key. Even with the best prompt engineering, LLMs can:

Hallucinate resource names and delete the wrong pod (or the right pod in the wrong namespace)
Misread ambiguous instructions — "scale down the api service" during an incident might mean the payment API, not the internal metrics API
Cascade failures — one command can trigger a reconciliation loop that touches dozens of dependent resources
Bypass audit trails — actions appear in the Kubernetes audit log under a service account, not attributed to the agent's reasoning chain

In a 273-system production fleet across AWS, Azure, and GCP, an unconstrained agent is not a productivity multiplier. It's a blast radius.

The Bouncer Pattern

The bouncer pattern sits an intermediary layer between your AI agent and the Kubernetes API server. The agent never talks to kubectl. It talks to the bouncer, which enforces four things:

Identity verification — which agent is this, and is it authorized for this class of operation?
Intent validation — does the requested action match an approved operation from a finite catalog?
Scope enforcement — can this agent touch this namespace, this resource type, this cluster?
Audit logging — every action is recorded with the full reasoning chain, not just the kubectl command

The bouncer can be a thin Kubernetes operator, a dedicated API gateway, or a structured tool layer in your agent framework. The implementation varies; the guarantee is the same — the agent gets a curated API, not raw cluster access.

The Bouncer architecture: agent → API gateway / operator → minimal RBAC service account → K8s API server

Step 1: Design the Allowed Operation Catalog

Before writing any code, define what your agent is actually allowed to do. Not "anything," not "read-only plus some writes." A finite, explicit list.

For a typical SRE automation agent, the catalog looks like:

# agent-operation-catalog.yaml
operations:
  # Safe reads — always allowed
  - id: get_pod_status
    verb: get
    resources: [pods]
    scope: any-namespace
  - id: get_deployment_status
    verb: get
    resources: [deployments, replicasets]
    scope: any-namespace
  - id: get_events
    verb: list
    resources: [events]
    scope: any-namespace

  # Controlled writes — require explicit approval or confidence threshold
  - id: restart_deployment
    verb: patch
    resources: [deployments]
    scope: non-production
    requires_confirmation: false
    max_replicas_affected: 3
  - id: scale_deployment
    verb: patch
    resources: [deployments]
    scope: any-namespace
    requires_confirmation: true # always confirm cross-env
    audit_level: high

  # Explicitly denied — no matter what the agent says
  - id: delete_namespace
    verb: delete
    resources: [namespaces]
    enabled: false
  - id: modify_clusterrolebinding
    verb: "*"
    resources: [clusterrolebindings, clusterroles]
    enabled: false

This catalog is the contract. The bouncer enforces it; the agent operates within it.

Step 2: Minimal-Privilege RBAC for the Agent Service Account

The bouncer uses a Kubernetes service account to perform its allowed operations. That account needs exactly the permissions the catalog defines — no more.

# agent-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sre-agent
  namespace: platform-system
  annotations:
    platform.io/agent-catalog: "sre-ops-v2"
    platform.io/managed-by: "agent-bouncer-operator"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sre-agent-role
rules:
  # Reads — all namespaces
  - apiGroups: [""]
    resources: ["pods", "events", "nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets"]
    verbs: ["get", "list", "watch"]

  # Controlled writes — deployments only, no deletes
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["patch", "update"]
  - apiGroups: ["apps"]
    resources: ["deployments/scale"]
    verbs: ["patch"]

  # Explicitly no access to
  # - namespaces (delete)
  # - clusterroles / clusterrolebindings
  # - secrets (use Vault instead)
  # - nodes (write)
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: sre-agent-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: sre-agent-role
subjects:
  - kind: ServiceAccount
    name: sre-agent
    namespace: platform-system

Key principle: if the operation isn't in the catalog, the service account can't do it even if the bouncer has a bug. RBAC is the last line of defense.

Step 3: OPA Gatekeeper — Policy-as-Code Enforcement

RBAC defines what resource types can be touched. OPA Gatekeeper adds semantic guardrails — policies about the state of resources, not just the type.

# constraint-no-agent-prod-delete.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: AgentOperationPolicy
metadata:
  name: no-agent-writes-in-prod
spec:
  enforcementAction: deny
  match:
    userInfo:
      username: "system:serviceaccount:platform-system:sre-agent"
    operations: ["UPDATE", "PATCH", "DELETE"]
    namespaceSelector:
      matchLabels:
        environment: production
  parameters:
    message: "SRE agent write operations in production require human confirmation. Use the agent API with requires_confirmation=true."

# policy: agent cannot scale below 1 replica
package agentops

violation[{"msg": msg}] {
  input.review.operation == "PATCH"
  input.review.object.kind == "Deployment"
  input.review.object.spec.replicas == 0
  msg := sprintf(
    "Agent cannot scale deployment %v to 0 replicas. Minimum is 1.",
    [input.review.object.metadata.name]
  )
}

Now, even if there's a bug in the bouncer logic and the agent constructs a malformed scale request, Gatekeeper catches it at admission time. Defense in depth.

Step 4: The Bouncer API — Structured Tools, Not Shell Commands

The agent's tool interface should be declarative and typed, not a shell escape hatch:

# tools/k8s_bouncer.py
from dataclasses import dataclass
from enum import Enum
from typing import Optional
import httpx

class OperationScope(str, Enum):
    READ = "read"
    WRITE_NON_PROD = "write_non_prod"
    WRITE_PROD_CONFIRMED = "write_prod_confirmed"

@dataclass
class AgentOperation:
    operation_id: str          # maps to catalog entry
    namespace: str
    resource_name: str
    parameters: dict
    reasoning: str             # agent explains WHY it's taking this action
    confidence: float          # 0.0-1.0

class K8sBouncer:
    def __init__(self, bouncer_url: str, agent_id: str, api_key: str):
        self.client = httpx.Client(
            base_url=bouncer_url,
            headers={"X-Agent-ID": agent_id, "X-API-Key": api_key},
            timeout=30.0,
        )

    def execute(self, op: AgentOperation) -> dict:
        """Submit an operation to the bouncer. Returns result or raises on denial."""
        response = self.client.post("/operations", json={
            "operation_id": op.operation_id,
            "namespace": op.namespace,
            "resource_name": op.resource_name,
            "parameters": op.parameters,
            "reasoning": op.reasoning,    # stored in audit log
            "confidence": op.confidence,
        })
        response.raise_for_status()
        return response.json()

    def get_pod_status(self, namespace: str, pod_name: str) -> dict:
        return self.execute(AgentOperation(
            operation_id="get_pod_status",
            namespace=namespace,
            resource_name=pod_name,
            parameters={},
            reasoning="Checking pod health during incident triage",
            confidence=1.0,
        ))

    def restart_deployment(
        self,
        namespace: str,
        deployment: str,
        reasoning: str,
        confidence: float,
    ) -> dict:
        if confidence < 0.85:
            raise ValueError(
                f"Confidence {confidence:.0%} is below the 85% threshold for restart operations. "
                "Escalate to on-call engineer."
            )
        return self.execute(AgentOperation(
            operation_id="restart_deployment",
            namespace=namespace,
            resource_name=deployment,
            parameters={"strategy": "rollout_restart"},
            reasoning=reasoning,
            confidence=confidence,
        ))

The agent never constructs a kubectl command. It calls typed methods with a reasoning field — because every action in the audit log must explain why the agent did it, not just what it did.

Step 5: Structured Audit Logging

Every bouncer operation should produce a structured audit event:

{
  "timestamp": "2025-11-10T14:23:11Z",
  "event_type": "agent_operation",
  "agent_id": "sre-agent-prod-01",
  "operation_id": "restart_deployment",
  "namespace": "payments",
  "resource": "payment-api",
  "outcome": "allowed",
  "confidence": 0.91,
  "reasoning": "OOMKilled events detected in last 3 pods. Memory limit appears to be undersized based on usage trend. Restarting to clear state while escalating resource limit review.",
  "k8s_audit_uid": "f3a92b1d-cc41-4e8a-9f47-0b32a8e12345",
  "policy_checks": [
    { "policy": "catalog_membership", "result": "pass" },
    { "policy": "rbac_pre_check", "result": "pass" },
    {
      "policy": "environment_scope",
      "result": "pass",
      "environment": "production",
      "requires_confirmation": false
    },
    { "policy": "confidence_threshold", "result": "pass", "threshold": 0.85, "actual": 0.91 }
  ],
  "duration_ms": 412
}

This isn't just for compliance. When the agent makes the wrong call (and it will), this log is the difference between a 2-hour post-mortem and a 2-minute root cause.

What the Bouncer Gives You

Bouncer audit dashboard — every agent decision logged with reasoning, confidence, and policy check results

After running this pattern across a 273-system production fleet, the numbers speak for themselves:

Zero unintended cluster mutations from AI agents in 6 months (previously: 3 incidents/quarter from misrouted automation)
92% of agent operations were GET/LIST reads — the catalog's write operations are rare and high-value
100% audit coverage — every agent action has a reasoning chain, not just a resource diff
< 200ms median bouncer overhead — the policy layer adds negligible latency for async SRE workflows
MTTR reduced by 34% on the incidents where the agent was involved — because it could act on read data without waiting for human approval

The agent didn't get slower. It got trustworthy.

Failure Modes to Design For

1. Confidence miscalibration — Your agent's confidence score might be wrong. Add a human-in-the-loop trigger at >= 80% of the max allowed replica impact, regardless of confidence.

2. Reasoning injection — An attacker who can influence the agent's context (log injection, alert body manipulation) might craft reasoning that bypasses soft guards. Hard RBAC and Gatekeeper policies are not optional.

3. Bouncer becoming a bottleneck — Deploy the bouncer as a replicated operator, not a single pod. SRE agents are often invoked during incidents — that's the worst time for the bouncer to be down.

4. Catalog drift — As your platform grows, agents request new operations. Treat catalog additions like production schema migrations: reviewed, versioned, and rolled out with a canary.

The Minimal Starting Point

You don't need to build a full operator on day one. Start here:

# 1. Create a minimal service account for the agent
kubectl create serviceaccount ai-agent -n platform-system

# 2. Grant read-only access to start
kubectl create clusterrolebinding ai-agent-readonly \
  --clusterrole=view \
  --serviceaccount=platform-system:ai-agent

# 3. Route all agent calls through a single wrapper that logs to stdout
# (Your SIEM picks it up; you have an audit trail from day one)

# 4. Graduate permissions one operation at a time,
#    measuring blast radius before each addition

Read-only first. One write permission at a time. Full audit logging from the start.

Conclusion

AI agents are genuinely useful for SRE work — alert correlation, runbook execution, incident summarization. But "useful" and "unconstrained" are incompatible with a production fleet.

The bouncer pattern lets you ship AI-assisted operations without gambling on an agent's judgment. The agent stays smart; the bouncer stays strict. That combination is what 99.99% SLA availability looks like when AI enters the on-call rotation.

Your cluster doesn't need to trust the agent. It needs to trust the bouncer.

Next Steps

Define your operation catalog before writing any agent code
Implement the bouncer as a Kubernetes operator using controller-runtime
Add OPA Gatekeeper constraints that mirror your catalog's deny list
Ship structured audit logs to your SIEM from day one
Graduate agent permissions quarterly — review what was actually used, remove what wasn't

The AI will get smarter. The guardrails should too.