Data processing: what happens to cluster data before and after the model¶

The agent does not pass raw cluster data to the model. Most data is read, filtered, grouped, summarised or cut in Python first, and the model's answers are checked and normalised before anything is stored or posted. This page lists every such step, and also what is not done, so that a security or privacy review can rely on it.

The flow:

 cluster / Prometheus / Alertmanager
        │  read (read-only; Secrets never requested)
        ▼
 ① reduce in Python      summaries, filters, grouping, formatting      (tools)
        ▼
 ② cap the size          every tool result ≤ TOOL_OUTPUT_MAX_CHARS      (agent loop)
        ▼
 ③ the request           system prompt + conversation + tool results ──▶ the model (any provider)
        ▼                                                                    │
 ④ check the answer      findings JSON parsed, validated, normalised  ◀─────┘
        ▼
 ⑤ compute facts         new / still open / resolved, the night's story  (memory, report)
        ▼
 ⑥ shape the output      Markdown report, Teams card shrunk to fit       (teams)

Plus one path where no data reaches the model at all: the "anything changed?" check of scheduled runs.

① Reduction in Python, per tool¶

For the morning report's history, two more reductions happen without the model: - Alerts that fired overnight come from one Prometheus query that already aggregates (max by (alertname, severity, namespace, pod, node, job)); Python then drops always-firing alerts (ALERTS_IGNORE_NAMES), info-level alerts and those still firing. - Findings history is computed from memory (below); only the result is sent.

② Size cap on every tool result¶

Whatever a tool returns, the agent loop cuts it to TOOL_OUTPUT_MAX_CHARS (default 8,000) before it is added to the conversation: two thirds from the head, one third from the tail, and a marker in between telling the model to narrow the query. The head of a listing has the columns and first rows; the tail of a log has the latest lines. This protects the context window and the cost: every later model call resends the whole conversation.

③ What the request to the model contains¶

The morning report's overview is a separate request without tools: the model gets only the facts computed in ⑤ (as Markdown) and writes 3–6 sentences. It cannot look anything up, so it can only summarise what Python gave it.

④ Checking the model's answer¶

The model's output is never trusted as-is:

• Findings JSON. The answer of a health check must end with a JSON block. It is parsed and validated: it must be an object with a findings list, and every finding must have a fingerprint. An unknown severity becomes warning (logged); a missing title falls back to the fingerprint. Invalid → one follow-up asking only for the JSON; still invalid → the run is recorded as failed and memory is not changed.
• Fingerprints are normalised (lower case, spaces around / removed, words joined with -), because the model does not always spell them the same way.
• Duplicate fingerprints in one answer are merged, keeping the most severe one.
• Learned notes (save_note): whitespace and newlines collapsed to one line; rejected if empty or longer than 500 characters; logged at WARNING.
• Tool calls: unknown tools, invalid JSON arguments and invalid values (namespace, event reason, time window, kubectl flags) are refused with ERROR: text before anything runs.

⑤ Facts computed in Python, not by the model¶

The model never decides what is "new" or "resolved"; it only reports what it sees now.

⑥ Shaping the output¶

• Markdown reports (stdout, history): rendered from the parsed findings and the facts, not copied from the model's free text (except the summary/overview).
• Teams cards: built from the same data; if larger than 27,000 bytes, details are shortened step by step and lists cut with "… and N more".
• Logs: tool calls are logged with their arguments and output size; full outputs only at DEBUG. The webhook URL is never logged.

What is NOT done¶

Being explicit matters here:

• No redaction or masking. Text that is passed through (logs from pods_log or kubectl logs, event messages, ConfigMap contents, annotations, describe output) reaches the model as it is. If an application writes personal data or a credential into its logs, the model can see it, and it can end up in an answer, the history and Teams. Secrets themselves are never read (RBAC, MCP denied resources, the kubectl guard).
• No anonymisation. Object names, namespaces, node names and IP addresses in messages are sent unchanged; the model needs them to investigate.
• No local filtering of what the model asks for. Within the read-only tools, the model chooses what to read (for example which pod's logs). Limits apply (tail, size cap), content filtering does not.

Mitigations: logs are read only when investigating a specific problem, with a tail; you choose the provider, so read its data and retention terms (list of links) or run the model inside your network (Ollama or another OpenAI-compatible server); the Responses API (Foundry, OpenAI) is called with store=false, so no conversation is stored by the provider; with Azure AI Foundry the model runs in a dedicated resource in the EU data zone and does not train on prompts; Secrets are out of reach; keep credentials out of logs and ConfigMaps (good practice anyway). If redaction is needed later, the natural place is the agent loop's step ②, where every tool result passes. See security.md.

Where this lives in the code¶