{"slug":"ai-agent-kubectl-bouncer","title":"Your AI Agent Doesn't Need kubectl. It Needs a Bouncer.","description":"Raw kubectl access for an AI agent is a cluster incident waiting to happen. Build a bouncer: a secure layer that lets agents act without running wild.","date":"2025-11-10","author":"Elaph Hilful","thumbnail":"/blog-images/k8s01.webp","tags":["Kubernetes","AI Agents","Security","RBAC","Platform Engineering","DevSecOps"],"category":"Security","featured":true,"readTime":10,"htmlContent":"

Every week another team shows me their new AI agent. It parses alerts, writes runbooks, even drafts incident post-mortems. Then they demo it live — and I watch the agent call kubectl delete pod on a production namespace.

\n

The room goes quiet.

\n

This is the pattern I keep seeing: teams build impressive AI capabilities and wire them directly to kubectl or the Kubernetes API server. No guardrails. No audit trail. No limits. The agent is brilliant — until it isn't, and then it has cluster-admin.

\n

The fix isn't to take away the agent's access. It's to give it a bouncer.

\n

\"AI

\n

The Problem with Direct kubectl Access

\n

A bouncer at a club doesn't care how charming you are. They check your ID, verify you're on the list, and turn you away if the venue is at capacity — no matter what you say. Your AI agent needs the same treatment.

\n

When agents call kubectl directly, you're handing them a master key. Even with the best prompt engineering, LLMs can:

\n\n

In a 273-system production fleet across AWS, Azure, and GCP, an unconstrained agent is not a productivity multiplier. It's a blast radius.

\n

The Bouncer Pattern

\n

The bouncer pattern sits an intermediary layer between your AI agent and the Kubernetes API server. The agent never talks to kubectl. It talks to the bouncer, which enforces four things:

\n
    \n
  1. Identity verification — which agent is this, and is it authorized for this class of operation?
    2. \n
  2. Intent validation — does the requested action match an approved operation from a finite catalog?
    3. \n
  3. Scope enforcement — can this agent touch this namespace, this resource type, this cluster?
    4. \n
  4. Audit logging — every action is recorded with the full reasoning chain, not just the kubectl command
    5. \n
\n

The bouncer can be a thin Kubernetes operator, a dedicated API gateway, or a structured tool layer in your agent framework. The implementation varies; the guarantee is the same — the agent gets a curated API, not raw cluster access.

\n

\"The

\n

Step 1: Design the Allowed Operation Catalog

\n

Before writing any code, define what your agent is actually allowed to do. Not \"anything,\" not \"read-only plus some writes.\" A finite, explicit list.

\n

For a typical SRE automation agent, the catalog looks like:

\n
# agent-operation-catalog.yaml\noperations:\n  # Safe reads — always allowed\n  - id: get_pod_status\n    verb: get\n    resources: [pods]\n    scope: any-namespace\n  - id: get_deployment_status\n    verb: get\n    resources: [deployments, replicasets]\n    scope: any-namespace\n  - id: get_events\n    verb: list\n    resources: [events]\n    scope: any-namespace\n\n  # Controlled writes — require explicit approval or confidence threshold\n  - id: restart_deployment\n    verb: patch\n    resources: [deployments]\n    scope: non-production\n    requires_confirmation: false\n    max_replicas_affected: 3\n  - id: scale_deployment\n    verb: patch\n    resources: [deployments]\n    scope: any-namespace\n    requires_confirmation: true # always confirm cross-env\n    audit_level: high\n\n  # Explicitly denied — no matter what the agent says\n  - id: delete_namespace\n    verb: delete\n    resources: [namespaces]\n    enabled: false\n  - id: modify_clusterrolebinding\n    verb: \"*\"\n    resources: [clusterrolebindings, clusterroles]\n    enabled: false\n
\n

This catalog is the contract. The bouncer enforces it; the agent operates within it.

\n

Step 2: Minimal-Privilege RBAC for the Agent Service Account

\n

The bouncer uses a Kubernetes service account to perform its allowed operations. That account needs exactly the permissions the catalog defines — no more.

\n
# agent-serviceaccount.yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: sre-agent\n  namespace: platform-system\n  annotations:\n    platform.io/agent-catalog: \"sre-ops-v2\"\n    platform.io/managed-by: \"agent-bouncer-operator\"\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n  name: sre-agent-role\nrules:\n  # Reads — all namespaces\n  - apiGroups: [\"\"]\n    resources: [\"pods\", \"events\", \"nodes\"]\n    verbs: [\"get\", \"list\", \"watch\"]\n  - apiGroups: [\"apps\"]\n    resources: [\"deployments\", \"replicasets\", \"statefulsets\"]\n    verbs: [\"get\", \"list\", \"watch\"]\n\n  # Controlled writes — deployments only, no deletes\n  - apiGroups: [\"apps\"]\n    resources: [\"deployments\"]\n    verbs: [\"patch\", \"update\"]\n  - apiGroups: [\"apps\"]\n    resources: [\"deployments/scale\"]\n    verbs: [\"patch\"]\n\n  # Explicitly no access to\n  # - namespaces (delete)\n  # - clusterroles / clusterrolebindings\n  # - secrets (use Vault instead)\n  # - nodes (write)\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: sre-agent-binding\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: sre-agent-role\nsubjects:\n  - kind: ServiceAccount\n    name: sre-agent\n    namespace: platform-system\n
\n

Key principle: if the operation isn't in the catalog, the service account can't do it even if the bouncer has a bug. RBAC is the last line of defense.

\n

Step 3: OPA Gatekeeper — Policy-as-Code Enforcement

\n

RBAC defines what resource types can be touched. OPA Gatekeeper adds semantic guardrails — policies about the state of resources, not just the type.

\n
# constraint-no-agent-prod-delete.yaml\napiVersion: constraints.gatekeeper.sh/v1beta1\nkind: AgentOperationPolicy\nmetadata:\n  name: no-agent-writes-in-prod\nspec:\n  enforcementAction: deny\n  match:\n    userInfo:\n      username: \"system:serviceaccount:platform-system:sre-agent\"\n    operations: [\"UPDATE\", \"PATCH\", \"DELETE\"]\n    namespaceSelector:\n      matchLabels:\n        environment: production\n  parameters:\n    message: \"SRE agent write operations in production require human confirmation. Use the agent API with requires_confirmation=true.\"\n
\n
# policy: agent cannot scale below 1 replica\npackage agentops\n\nviolation[{\"msg\": msg}] {\n  input.review.operation == \"PATCH\"\n  input.review.object.kind == \"Deployment\"\n  input.review.object.spec.replicas == 0\n  msg := sprintf(\n    \"Agent cannot scale deployment %v to 0 replicas. Minimum is 1.\",\n    [input.review.object.metadata.name]\n  )\n}\n
\n

Now, even if there's a bug in the bouncer logic and the agent constructs a malformed scale request, Gatekeeper catches it at admission time. Defense in depth.

\n

Step 4: The Bouncer API — Structured Tools, Not Shell Commands

\n

The agent's tool interface should be declarative and typed, not a shell escape hatch:

\n
# tools/k8s_bouncer.py\nfrom dataclasses import dataclass\nfrom enum import Enum\nfrom typing import Optional\nimport httpx\n\nclass OperationScope(str, Enum):\n    READ = \"read\"\n    WRITE_NON_PROD = \"write_non_prod\"\n    WRITE_PROD_CONFIRMED = \"write_prod_confirmed\"\n\n@dataclass\nclass AgentOperation:\n    operation_id: str          # maps to catalog entry\n    namespace: str\n    resource_name: str\n    parameters: dict\n    reasoning: str             # agent explains WHY it's taking this action\n    confidence: float          # 0.0-1.0\n\nclass K8sBouncer:\n    def __init__(self, bouncer_url: str, agent_id: str, api_key: str):\n        self.client = httpx.Client(\n            base_url=bouncer_url,\n            headers={\"X-Agent-ID\": agent_id, \"X-API-Key\": api_key},\n            timeout=30.0,\n        )\n\n    def execute(self, op: AgentOperation) -> dict:\n        \"\"\"Submit an operation to the bouncer. Returns result or raises on denial.\"\"\"\n        response = self.client.post(\"/operations\", json={\n            \"operation_id\": op.operation_id,\n            \"namespace\": op.namespace,\n            \"resource_name\": op.resource_name,\n            \"parameters\": op.parameters,\n            \"reasoning\": op.reasoning,    # stored in audit log\n            \"confidence\": op.confidence,\n        })\n        response.raise_for_status()\n        return response.json()\n\n    def get_pod_status(self, namespace: str, pod_name: str) -> dict:\n        return self.execute(AgentOperation(\n            operation_id=\"get_pod_status\",\n            namespace=namespace,\n            resource_name=pod_name,\n            parameters={},\n            reasoning=\"Checking pod health during incident triage\",\n            confidence=1.0,\n        ))\n\n    def restart_deployment(\n        self,\n        namespace: str,\n        deployment: str,\n        reasoning: str,\n        confidence: float,\n    ) -> dict:\n        if confidence < 0.85:\n            raise ValueError(\n                f\"Confidence {confidence:.0%} is below the 85% threshold for restart operations. \"\n                \"Escalate to on-call engineer.\"\n            )\n        return self.execute(AgentOperation(\n            operation_id=\"restart_deployment\",\n            namespace=namespace,\n            resource_name=deployment,\n            parameters={\"strategy\": \"rollout_restart\"},\n            reasoning=reasoning,\n            confidence=confidence,\n        ))\n
\n

The agent never constructs a kubectl command. It calls typed methods with a reasoning field — because every action in the audit log must explain why the agent did it, not just what it did.

\n

Step 5: Structured Audit Logging

\n

Every bouncer operation should produce a structured audit event:

\n
{\n  \"timestamp\": \"2025-11-10T14:23:11Z\",\n  \"event_type\": \"agent_operation\",\n  \"agent_id\": \"sre-agent-prod-01\",\n  \"operation_id\": \"restart_deployment\",\n  \"namespace\": \"payments\",\n  \"resource\": \"payment-api\",\n  \"outcome\": \"allowed\",\n  \"confidence\": 0.91,\n  \"reasoning\": \"OOMKilled events detected in last 3 pods. Memory limit appears to be undersized based on usage trend. Restarting to clear state while escalating resource limit review.\",\n  \"k8s_audit_uid\": \"f3a92b1d-cc41-4e8a-9f47-0b32a8e12345\",\n  \"policy_checks\": [\n    { \"policy\": \"catalog_membership\", \"result\": \"pass\" },\n    { \"policy\": \"rbac_pre_check\", \"result\": \"pass\" },\n    {\n      \"policy\": \"environment_scope\",\n      \"result\": \"pass\",\n      \"environment\": \"production\",\n      \"requires_confirmation\": false\n    },\n    { \"policy\": \"confidence_threshold\", \"result\": \"pass\", \"threshold\": 0.85, \"actual\": 0.91 }\n  ],\n  \"duration_ms\": 412\n}\n
\n

This isn't just for compliance. When the agent makes the wrong call (and it will), this log is the difference between a 2-hour post-mortem and a 2-minute root cause.

\n

What the Bouncer Gives You

\n

\"Bouncer

\n

After running this pattern across a 273-system production fleet, the numbers speak for themselves:

\n\n

The agent didn't get slower. It got trustworthy.

\n

Failure Modes to Design For

\n

1. Confidence miscalibration — Your agent's confidence score might be wrong. Add a human-in-the-loop trigger at >= 80% of the max allowed replica impact, regardless of confidence.

\n

2. Reasoning injection — An attacker who can influence the agent's context (log injection, alert body manipulation) might craft reasoning that bypasses soft guards. Hard RBAC and Gatekeeper policies are not optional.

\n

3. Bouncer becoming a bottleneck — Deploy the bouncer as a replicated operator, not a single pod. SRE agents are often invoked during incidents — that's the worst time for the bouncer to be down.

\n

4. Catalog drift — As your platform grows, agents request new operations. Treat catalog additions like production schema migrations: reviewed, versioned, and rolled out with a canary.

\n

The Minimal Starting Point

\n

You don't need to build a full operator on day one. Start here:

\n
# 1. Create a minimal service account for the agent\nkubectl create serviceaccount ai-agent -n platform-system\n\n# 2. Grant read-only access to start\nkubectl create clusterrolebinding ai-agent-readonly \\\n  --clusterrole=view \\\n  --serviceaccount=platform-system:ai-agent\n\n# 3. Route all agent calls through a single wrapper that logs to stdout\n# (Your SIEM picks it up; you have an audit trail from day one)\n\n# 4. Graduate permissions one operation at a time,\n#    measuring blast radius before each addition\n
\n

Read-only first. One write permission at a time. Full audit logging from the start.

\n

Conclusion

\n

AI agents are genuinely useful for SRE work — alert correlation, runbook execution, incident summarization. But \"useful\" and \"unconstrained\" are incompatible with a production fleet.

\n

The bouncer pattern lets you ship AI-assisted operations without gambling on an agent's judgment. The agent stays smart; the bouncer stays strict. That combination is what 99.99% SLA availability looks like when AI enters the on-call rotation.

\n

Your cluster doesn't need to trust the agent. It needs to trust the bouncer.

\n

Next Steps

\n\n

The AI will get smarter. The guardrails should too.

"}