Model providers¶

k8s-servant works with the model you already have: Azure AI Foundry, OpenAI, Anthropic (Claude), a self-hosted model through Ollama, or any other server that speaks the OpenAI Chat Completions API (vLLM, LiteLLM, LM Studio, Gemini, ...). You choose the provider with LLM_PROVIDER; this page explains the choice and gives a complete recipe for each one: when to pick it, the .env for your laptop, the Helm values file for a cluster, how to supply the key (or set up Entra ID), and what doctor should print.

No provider is the "main" one. Foundry is listed first only because the first release supported nothing else.

• Why three APIs
• Choose a provider
• Foundry (GPT and other Responses models)
• Claude on Foundry
• OpenAI
• Anthropic
• Ollama
• OpenAI-compatible (vLLM, LiteLLM, Gemini, ...)
• Settings each provider ignores
• Migrating from FOUNDRY_* settings

Why three APIs¶

An API here is the set of rules for sending a request to a model and reading its reply. The agent speaks three of them, one per family of providers. You never choose an API directly: the provider decides.

The reason there is more than one is how newer models work. A reasoning model thinks in hidden steps before it answers. The agent works in a loop: the model asks for a tool (for example "list the pods"), the agent runs it and sends the result back, and the model asks for the next tool. In that loop the model needs its own hidden steps from the previous turn, otherwise it forgets why it asked for the tool and works worse.

The older Chat Completions API has no place to carry those hidden steps from one tool call to the next. The newest GPT models therefore refuse to combine tools with reasoning on it, and the exact cut-off differs by platform:

• OpenAI API: "Starting with GPT-5.4, Chat Completions does not support tool calling with reasoning_effort values other than none" (OpenAI: migrate to Responses).
• Microsoft Foundry: the gpt-5.6 models "support the Chat Completions API and tools, but can't combine reasoning with tools on Chat Completions". A request with tools fails even without reasoning_effort, because these models default to medium. gpt-5.4 and gpt-5.5 still allow it on Foundry (Microsoft Learn).

So an older deployment such as gpt-5.4-mini on Foundry works on Chat Completions (the agent's first release ran that way), but moving to the Responses API keeps current and future models working. The two newer APIs do have a place for the hidden steps. They hand the hidden steps back in a sealed (encrypted or signed) form that the agent keeps unread and sends back unchanged on the next call:

Anthropic also offers an OpenAI-compatible endpoint, but it describes it as not meant for production use (reasoning effort is ignored, prompt caching is unsupported). That is why Claude uses the Messages API.

The rest of the agent (the loop, health checks, reports, memory) does not know which API is behind it.

Choose a provider¶

Rules that hold for every provider:

• LLM_MODEL is required. On Foundry it is the deployment name.
• LLM_AUTH is auto (default), entra or api-key. auto uses a key when one is configured, otherwise Entra ID where the endpoint is a Foundry address, otherwise no sign-in for ollama and openai-compatible, otherwise it refuses to start. entra and api-key are strict: entra refuses to start if a key is also set, api-key refuses to start without one.
• Entra ID is only ever used for Foundry endpoints, so its tokens can never reach a third-party server. A Foundry endpoint is a host ending in .openai.azure.com, .services.ai.azure.com or .cognitiveservices.azure.com. These hosts are always https.
• With auto, a configured key wins over Workload Identity. doctor then prints a warning; set LLM_AUTH=entra and remove the key to use Workload Identity.
• Keys are read from LLM_API_KEY_FILE (a file; in the cluster the chart mounts your Secret there) or LLM_API_KEY; the file wins. The key is never logged.
• Entra ID needs the Azure extra: pip install 'k8s-servant[azure]', or pip install -e '.[dev,azure]' in a clone. The container image already includes it.
• LLM_REASONING_EFFORT defaults to medium for foundry, openai and anthropic, and to empty (not sent) for ollama and openai-compatible. On Anthropic, minimal is sent as low (with a warning in the log, as the Messages API has no minimal) and none turns thinking off.
• LLM_MAX_OUTPUT_TOKENS (default 32000, thinking included) is sent only on the Messages API, which requires a limit. A reply that is cut off by a limit is an error and the run is recorded as failed, as is a failed, cancelled or content-filtered reply. Raise LLM_MAX_OUTPUT_TOKENS or lower the effort.
• temperature is never sent (reasoning models reject it).
• The full settings table is in configuration.md. To start from a ready-made Helm values file, copy one from examples/ (listed in each recipe).

Every recipe below installs with plain Helm. Flux or another GitOps tool works the same way: see examples/flux-helmrelease.yaml if you prefer it.

Foundry (GPT and other Responses models)¶

Choose it when your organisation runs on Azure and you want a model inside your own tenant, in an EU or US data zone, with no key to manage: on AKS the pod signs in with Workload Identity. It serves GPT models and others (for example DeepSeek, Llama, Grok) through the Responses API. Cognitive Services OpenAI User, the role the setup script assigns by default, covers only OpenAI models: for DeepSeek, Llama, Grok and other non-OpenAI models run the script with MODEL_ROLE="Cognitive Services User" (Microsoft Learn). Partner models that offer only Chat Completions use openai-compatible with a Foundry endpoint, and Claude has its own recipe.

The Azure side (resource, deployment, roles, network) is in foundry-setup.md and the background in foundry-concepts.md; scripts/setup-foundry.sh does it in one go.

Laptop .env (sign in with az login; no key; needs the azure extra):

LLM_PROVIDER=foundry
LLM_ENDPOINT=https://my-foundry.openai.azure.com/   # /openai/v1/ is added for you
LLM_MODEL=my-deployment                              # the deployment name
LLM_AUTH=entra

Cluster: examples/values-azure.yaml (Workload Identity, Teams webhook synced from Key Vault by External Secrets) or examples/values-azure-csi.yaml (no Kubernetes Secrets at all). The relevant part:

azureWorkloadIdentity:
  enabled: true
  clientId: "00000000-0000-0000-0000-000000000000"
  tenantId: "00000000-0000-0000-0000-000000000000"
appConfig:
  CLUSTER_NAME: my-cluster
  LLM_PROVIDER: foundry
  LLM_ENDPOINT: https://my-foundry.openai.azure.com/openai/v1/
  LLM_MODEL: my-deployment
  LLM_AUTH: entra

Entra ID setup: run scripts/setup-workload-identity.sh (identity, federated credential, the Cognitive Services OpenAI User role on the Foundry resource), put the client and tenant IDs it prints into azureWorkloadIdentity, then follow foundry-setup.md. With LLM_AUTH=entra no key is ever needed, and you can set disableLocalAuth: true on the resource so keys cannot be used at all.

If you must use a key instead (a quick experiment, or another tenant), create the Secret as in OpenAI, set LLM_AUTH: api-key and llmApiKey.existingSecret.

Expected doctor model line:

model: foundry / my-deployment via Responses API, Entra ID (Workload Identity)

On a laptop the source reads Entra ID (az login / DefaultAzureCredential).

Claude on Foundry¶

Choose it when you want Claude but inside Azure: same Entra ID sign-in and billing as the Foundry recipe, with Claude's Messages API. Deploy a Claude model in your Foundry resource first (foundry-setup.md, Step 2, choose a Claude model).

Laptop .env:

LLM_PROVIDER=anthropic
LLM_ENDPOINT=https://my-foundry.services.ai.azure.com   # /anthropic is added for you
LLM_MODEL=claude-sonnet-5-5                              # the deployment name
LLM_AUTH=entra

Cluster: examples/values-azure-claude.yaml.

azureWorkloadIdentity:
  enabled: true
  clientId: "00000000-0000-0000-0000-000000000000"
  tenantId: "00000000-0000-0000-0000-000000000000"
appConfig:
  CLUSTER_NAME: my-cluster
  LLM_PROVIDER: anthropic
  LLM_ENDPOINT: https://my-foundry.services.ai.azure.com
  LLM_MODEL: claude-sonnet-5-5
  LLM_AUTH: entra

Entra ID setup is the same as for Foundry, except for the role: Cognitive Services OpenAI User covers only OpenAI models, and Claude needs Cognitive Services User (Microsoft Learn). Run MODEL_ROLE="Cognitive Services User" scripts/setup-workload-identity.sh ..., then put the identity's client and tenant IDs in azureWorkloadIdentity. Entra ID is accepted for anthropic only because the endpoint is a Foundry address.

Expected doctor model line:

model: anthropic / claude-sonnet-5-5 via Messages API, Entra ID (Workload Identity)

OpenAI¶

Choose it when you have an OpenAI account (a ChatGPT subscription is not an API account) and want the shortest path, on a laptop or in any cluster. It uses the Responses API.

Laptop .env:

LLM_PROVIDER=openai
LLM_MODEL=gpt-5
LLM_API_KEY_FILE=/path/to/openai-api-key   # or LLM_API_KEY=... for a quick test

No LLM_ENDPOINT is needed (it defaults to https://api.openai.com/v1).

Cluster: examples/values-openai.yaml. Put the key in a file named api-key and create the Secret (this keeps it out of your shell history):

kubectl create namespace k8s-servant
kubectl -n k8s-servant create secret generic k8s-servant-llm --from-file=./api-key
rm ./api-key
helm install k8s-servant oci://ghcr.io/k8s-servant/charts/k8s-servant \
  -n k8s-servant -f examples/values-openai.yaml

appConfig:
  CLUSTER_NAME: my-cluster
  LLM_PROVIDER: openai
  LLM_MODEL: gpt-5
  LLM_AUTH: api-key
llmApiKey:
  existingSecret: k8s-servant-llm

The chart mounts the Secret as the file /var/run/secrets/llm/api-key and sets LLM_API_KEY_FILE for you; the key is never an environment variable. External Secrets, Sealed Secrets or the Secrets Store CSI driver work just as well, as long as a Secret or file with the key exists (deployment.md).

Expected doctor model line:

model: openai / gpt-5 via Responses API, API key (file /var/run/secrets/llm/api-key)

Anthropic¶

Choose it when you want Claude directly from Anthropic, on a laptop or in any cluster, with prompt caching (the large system prompt is repeated on every run, so caching saves cost and time). It uses the Messages API.

Laptop .env:

LLM_PROVIDER=anthropic
LLM_MODEL=claude-sonnet-5-5
LLM_API_KEY_FILE=/path/to/anthropic-api-key   # or LLM_API_KEY=... for a quick test

No LLM_ENDPOINT is needed (it defaults to https://api.anthropic.com).

Cluster: examples/values-anthropic.yaml. Create the Secret exactly as for OpenAI (a file named api-key, secret k8s-servant-llm), then:

appConfig:
  CLUSTER_NAME: my-cluster
  LLM_PROVIDER: anthropic
  LLM_MODEL: claude-sonnet-5-5
  LLM_AUTH: api-key
llmApiKey:
  existingSecret: k8s-servant-llm

Expected doctor model line:

model: anthropic / claude-sonnet-5-5 via Messages API, API key (file /var/run/secrets/llm/api-key)

Reasoning effort maps onto Anthropic's effort levels (minimal is sent as low with a warning in the log, none switches thinking off). If a reply hits the output limit the run fails with a message naming LLM_MAX_OUTPUT_TOKENS.

Ollama (laptop and in-cluster)¶

Choose it when the cluster data must not leave your network, or you want to try the agent for free. Quality depends on the model: it must support tool calling (for example qwen3), and small models find fewer problems than the hosted ones. It uses the Chat Completions API, needs no key, and does not send a reasoning effort.

Laptop (Ollama running locally):

ollama pull qwen3

LLM_PROVIDER=ollama
LLM_MODEL=qwen3
# LLM_ENDPOINT defaults to http://localhost:11434/v1

In the cluster: run Ollama as a Deployment with a Service (any way you like; the model must be pulled into it), then use examples/values-ollama.yaml:

appConfig:
  CLUSTER_NAME: my-cluster
  LLM_PROVIDER: ollama
  LLM_ENDPOINT: http://ollama.ollama.svc:11434/v1
  LLM_MODEL: qwen3

Plain http is accepted for localhost and for in-cluster hosts (a name without dots, *.svc or *.svc.cluster.local), by design: a model in the same cluster is the main use case. Anything else must be https.

Expected doctor model line:

model: ollama / qwen3 via Chat Completions API, no key

OpenAI-compatible: vLLM, LiteLLM, Gemini and others¶

Choose it when your server speaks the OpenAI Chat Completions API and is not one of the providers above: self-hosted vLLM or LM Studio, a LiteLLM gateway in front of many models, Google Gemini's OpenAI-compatible endpoint, or a partner model on Foundry that only offers Chat Completions. LLM_ENDPOINT is the full base URL including its path (usually ending in /v1). A key is sent only if you configure one. The reasoning effort is not sent unless you set LLM_REASONING_EFFORT, because the model behind the endpoint is unknown and many reject it. The model must support tool calling.

Cluster base: examples/values-minimal.yaml is exactly this provider with an API key and a Teams webhook:

appConfig:
  CLUSTER_NAME: my-cluster
  LLM_PROVIDER: openai-compatible
  LLM_ENDPOINT: https://api.example.com/v1
  LLM_MODEL: my-model
  LLM_AUTH: api-key
llmApiKey:
  existingSecret: llm-api-key

Create the key Secret from a file named api-key: kubectl -n k8s-servant create secret generic llm-api-key --from-file=./api-key. For an endpoint without a key, drop LLM_AUTH and llmApiKey. Worked examples:

vLLM (in the cluster, no key):

LLM_PROVIDER=openai-compatible
LLM_ENDPOINT=http://vllm.inference.svc:8000/v1
LLM_MODEL=Qwen/Qwen3-32B

Start vLLM with tool calling enabled (--enable-auto-tool-choice and a --tool-call-parser for the model); without it the agent gets no tool calls.

LiteLLM gateway (with a virtual key):

LLM_PROVIDER=openai-compatible
LLM_ENDPOINT=https://litellm.example.com/v1
LLM_MODEL=gpt-5            # a model name configured in LiteLLM
LLM_API_KEY_FILE=/path/to/litellm-key

Gemini:

LLM_PROVIDER=openai-compatible
LLM_ENDPOINT=https://generativelanguage.googleapis.com/v1beta/openai/
LLM_MODEL=gemini-2.5-pro
LLM_API_KEY_FILE=/path/to/gemini-key

Partner model on Foundry that offers only Chat Completions: use LLM_ENDPOINT=https://my-foundry.services.ai.azure.com/openai/v1/ (or your resource's address) and LLM_AUTH=entra; Entra ID is accepted because the endpoint is a Foundry address.

Expected doctor model lines:

model: openai-compatible / my-model via Chat Completions API, API key (file /var/run/secrets/llm/api-key)
model: openai-compatible / Qwen/Qwen3-32B via Chat Completions API, no key

Settings each provider ignores¶

A setting a provider has no use for has no effect; the agent does not warn about it.

Migrating from FOUNDRY_* settings¶

The settings of the single-provider releases were renamed, with no aliases. If an old name is still set, the agent refuses to start and names the new one. The Helm chart refuses an old appConfig key the same way. Also set LLM_PROVIDER: the app requires it, and you use foundry for what you had before. The chart's values.yaml already sets LLM_PROVIDER: foundry, so chart users on any other provider must override it.

The credential settings do not map to appConfig keys: the chart rejects FOUNDRY_API_KEY and FOUNDRY_API_KEY_FILE in appConfig. They become llmApiKey.existingSecret (or llmApiKey.file), which the chart turns into LLM_API_KEY_FILE.

New settings: LLM_PROVIDER, LLM_AUTH and LLM_MAX_OUTPUT_TOKENS. In Helm values the keys under appConfig are renamed the same way; llmApiKey is unchanged.

Data handling and errors¶

• The Responses API is called with store=false: OpenAI and Foundry do not keep the conversation. For Anthropic and every other provider, read the provider's own data and retention terms: Anthropic privacy centre, OpenAI API data controls, Gemini API terms. More in data-processing.md and security.md.
• Errors name the setting to fix and never contain the key. Foundry gets hints about role assignment, private endpoints and deployment names; other providers get generic ones (key rejected, unknown model, rate limit, cannot reach LLM_ENDPOINT). See troubleshooting.md.