Whitepaper
Mindbreeze InSpire LLM: A Kubernetes Integration Guide

Introduction

Mindbreeze InSpire LLM is a Kubernetes-native platform for deploying, exposing, and governing large language models and related inference services inside your own infrastructure. Instead of relying on external, hosted providers, it lets you run open-weight and self-hosted models close to your data – under you own security, network, and cost controls – while presenting a consistent, OpenAI-compatible API to consumers such as the InSpire RAG service.

The platform is built from three cooperating components:

mindbreeze-llm-inference – deploys the model workloads themselves. It supports multiple inference engines (Hugging Face TGI, vLLM, SGLang, and Speaches AI for speech-to-text) and can serve text, multimodal, OCR, and STT models on GPU and CPU. Models are pulled directly from the Hugging Face Hub by their model ID.
mindbreeze-api-llm-proxy – the API gateway in front of the models. It centralizes authentication, role-based access control, rate limiting, cost management, organization/team/user impersonation, and webhook-based event forwarding.
mindbreeze-llm-proxy – the ingress tier that terminates and routes external traffic to the API proxy and inference services.

Around these, InSpire LLM adds enterprise controls: Keycloak-based authentication, Kubernetes Network Policies for strict traffic isolation, and Helm/ArgoCD-driven deployment for repeatable rollouts. The result is a single, self-hosted stack covering the full lifecycle – discovering and deploying a model, exposing it securely, controlling who may use it and how much, and feeding usage data back into your own systems.

Chart Template

Mindbreeze LLM Inference

Mindbreeze LLM Inference allows an easy way to deploy LLM models using Hugging Face TGI, vLLM, SGLang and Speaches AI engines.

Deployment

Add Mindbreeze LLM Inference as a dependency in your Chart.yaml:

- name: mindbreeze-llm-inference

version: <latest-helm-chart-version>

repository: (alias) or (oci resource)

alias: default-text-llm

Note:

An alias is a changeable key and it should be equal to the endpoint in the final URL structure that would be exposed to the user.
To ensure a zero-downtime update, make sure to deploy LLM with two pods.
Each dependency is a separate deployment, so you may define multiple mindbreeze-llm-inference dependencies if you need to deploy multiple LLMs. Just keep in mind the resource constraints you are dealing with.

Configuration

Configure the LLM inference service under its alias in the values.yaml.

Example Configuration

default-text-llm: # alias

enabled: true

instanceCount: 2

image:

repository: "<repository>"

baseName: "<base-name>"

version: "<latest-version>"

registryPullSecrets: <registry-pull-secrets>

runtime: “tgi”

model:

id: "meta-llama/Llama-3.1-8B-Instruct"

maxTotalTokens: 24576

hub:

tokenSecret: hf-token-secret

env:

value: <http_proxy>

value: <https_proxy>

gpu:

enabled: true

profile: "nvidia.com/mig-4g.40gb"

resourcesMap:

storage: 100Gi

requests:

cpu: 5

memory: 20Gi

limits:

cpu: 10

memory: 50Gi

storageClassName: <storage-class-name>

readinessProbe:

initialDelaySeconds: 120

periodSeconds: 60

failureThreshold: 15

livenessProbe:

initialDelaySeconds: 120

periodSeconds: 60

failureThreshold: 15

Essential Configuration Elements

Most of the configuration elements don’t need to be changed. You can find all of the available configuration options in Appendix .

Mindbreeze LLM Proxy

Mindbreeze LLM Proxy is a reverse proxy for LLM deployments. All client requests first go to the proxy, which then routes them to the appropriate LLM service. Mindbreeze LLM Proxy exposes LLM services externally.

Deployment

Add Mindbreeze LLM Proxy as a dependency in your Chart.yaml:

- name: mindbreeze-llm-proxy

version: <latest-helm-chart-version>

repository: (alias:<alias>) or (oci resource)

alias: llm-proxy

Note:

A single proxy deployment with two pods is sufficient for zero-downtime updates.
The proxy operates using a rolling update strategy.

Configuration

Configure the proxy under its alias in the values.yaml.

Example Configuration

llm-proxy: # alias

instanceCount: 2

image:

repository: "<repository>"

baseName: "<base-name>"

version: " <latest-version>"

registryPullSecrets: <registry-pull-secrets>

clusterExternalDomain: <cluster-external-domain>

ingressTLSSecret: <ingress-tls-secret>

endpoint: llms

apiProxy:

charts:

api-llm-proxy

models:

- model: default-text-llm

charts:

- default-text-llm

limits:

workerConnections: 1024

connectionsPerUser: 4 # Rate limit (per user)

connectionsGlobal: 20 # Rate limit (globally)

Essential Configuration Elements

Most of the configuration elements don’t need to be changed. You can find all of the available configuration options in Appendix .

URL Structure

The exposed URL follows this template:

https://<endpoint>.<clusterExternalDomain>/api/llm/<user>/<models.model>/<api>

For example, a user might access an LLM from an example above using this URL:

https://llms.apps.openshift.com/api/llm/mindbreeze-user/default-text-llm/<api>

Limits – Default Values

Connection limit per user: 4
Global connection limit: 20
Worker connections: 1024

Creating a Secret

To deploy an LLM, it needs to be downloaded from a Huggingface hub. For, this a token needs to be set up. It is recommended to deploy it as a secret by an admin.

The secret should look the following way:

apiVersion: v1

kind: Secret

metadata:

type: Opaque

stringData:

token: "<token>"

In the token placeholder, you should place your Huggingface token. In addition, the name key is changeable, however the change should also be made in the values.yaml part:

hub:

tokenSecret: <new-secret-name>

Authentication

As of the Mindbreeze InSpire 25.4 Release, Mindbreeze Inspire LLM supports authentication and role-based access control for LLM usage. Authentication is handled with Keycloak, and should be configured the proper way.

Keycloak

To enable authentication, you must configure Keycloak with:

A Realm
A Client
Realm Roles

While most configurations are flexible, the realm roles must match the expected values.

Keycloak Configuration Example

This section provides a step-by-step example of a minimal Keycloak setup for Mindbreeze InSpire LLM.

Creating a Realm

Log in to the Keycloak Administration Console.
In the left-hand navigation pane, select “Realm” -> “Create Realm”.
Enter a “Realm name” for the realm (e.g., mb-inspire-llm) and click “Create”.

Creating a client

Within your newly created realm, navigate to “Clients” -> “Create client”.
Set the “Client type” to OpenID Connect.
Enter a “Client ID” (e.g., mb-inspire-llm-client).
Enable “Client authentication”.
Enable the following “Authentication flow” options:
- Standard flow
- Direct access grants
- Service accounts roles

Click “Save”.

Assigning roles to a client

Navigate to “Realm roles” -> “Create role”.
Enter a “Role name”. It should be equal to ”InSpire LLM Services User”.
You can create additional roles to enable role-based access control for LLM usage later.
Click “Save”.

Once role(s) are created, navigate to “Clients” -> “<your-created-client>” -> “Service accounts roles”.
Click “Assign role”. Filter by realm roles and select “InSpire LLM Services User”. Click “Assign”.

Once all of these steps are done, authentication can be enabled for LLM deployments.

Authentication Proxy Deployment

Authentication is facilitated via a dedicated proxy, which serves as an intermediary between the main proxy and the LLM deployment. Below is an example and a template on how to correctly configure a proxy for authentication.

Example: Deploying Authentication proxy

Add dependencies

First step is to add a new dependency for the proxy to the Chart.yaml:

apiVersion: v2

description: MB LLM Deployment

version: 0.0.1

dependencies:

- name: mindbreeze-llm-inference# vLLM deployment

version: <latest-helm-chart-version>

repository: alias:mindbreeze

alias: default-text-llm-vllm

condition: default-text-llm-vllm.enabled

- name: mindbreeze-llm-inference# Huggingface TGI deployment

version: <latest-helm-chart-version>

repository: alias:mindbreeze

alias: default-text-llm-tgi

condition: default-text-llm-tgi.enabled

- name: mindbreeze-llm-proxy# LLM proxy deployment

version: <latest-helm-chart-version>

repository: alias:mindbreeze

alias: llm-proxy

condition: llm-proxy.enabled

- name: mindbreeze-api-llm-proxy# Authentication proxy deployment

version: <latest-helm-chart-version>

repository: alias:mindbreeze

alias: api-llm-proxy

condition: api-llm-proxy.enabled

Proxy configuration

Example configuration for the LLM and authentication proxy:

llm-proxy:

enabled: true

...

models:

- model: default-text-llm-vllm

charts:

- default-text-llm-vllm

- model: default-text-llm-tgi

charts:

- default-text-llm-tgi

apiProxy:

charts:

api-llm-proxy

...

apiProxy.charts must match the alias of your authentication proxy.

Authentication Proxy Configuration

api-llm-proxy:

enabled: true

instanceCount: 1

image:

repository: "<repository>"

baseName: "<baseName>"

registryPullSecrets: “<registryPullSecrets>”

keycloak:

secret: keycloak-secret

realmKey: realm

urlKey: url

models:

- model: default-text-llm-vllm

model_id: unsloth/Qwen3-32B-unsloth-bnb-4bit

chart: default-text-llm-vllm

requests_per_minute: 100

- model: default-text-llm-tgi

model_id: meta-llama/Llama-3.1-8B-Instruct

chart: default-text-llm-tgi

requests_per_minute: 100

required_roles: ["Meta Llama Model Access"]

Configuration Parameters

You can find all of the available configuration options in Appendix .

Creating a Keycloak Configuration

To deploy an API Proxy, you need to setup a Keycloak configuration either in a ConfigMap or a Secret.

ConfigMap

The ConfigMap should look the following way:

apiVersion: v1

kind: ConfigMap

metadata:

data:

realm: "<keycloak-realm>"

url: "<keycloak-auth-url>"

In API Proxy section of values.yaml, you then need to update the following fields, which correspond to your ConfigMap data:

keycloak:

useConfigMap: true

configMap: keycloak-cm

realmKey: realm

urlKey: url

Secret

The Secret should look the following way:

apiVersion: v1

kind: Secret

metadata:

type: Opaque

stringData:

realm: "<keycloak-realm>"

url: "<keycloak-auth-url>"

In API Proxy section of values.yaml, you then need to update the following fields, which correspond to your secret data:

keycloak:

secret: keycloak-secret

realmKey: realm

urlKey: url

RAG Service Integration

Once Keycloak and the authentication proxy are configured, LLMs can be securely used within the RAG service.

Creating credentials

First step is to configure a credential and an endpoint in InSpire to automatically get an access token with every request to InSpire LLM deployment.

To configure a credential:

Log in to the Mindbreeze InSpire Management Center.
In the left-hand navigation pane, select “Configuration” -> “Network”.
In the “Credentials” section, click “Add Credential”.
Enter the following details:
- Name: <custom-credential-name> (e.g., InSpire LLM Credential)
- Type: OAuth 2
- Access Token URL:
- Realm: <realm>
- Client ID: <client-id>
- Client Secret: <client-secret>

In the “Endpoints” section, click “Add endpoint”.
Enter the following details:
- Location: <base-url-of-the-InSpireLLM>
- Credential: <credential-you-have-created-above>

Click “Save”.

Creating an LLM in RAG service

The URL to access the LLM service follows this pattern:

https://<endpoint>.<clusterExternalDomain>/api/llm/<user>/<models.model>/<llm-api>

Authentication

The format of the <user> segment in the URL depends on the authentication method:

Client Credentials Authentication:

When authenticating with client credentials, the <user> must follow this format:

service-account-<client-id>

Username/Password Authentication:

When authenticating with a username and password, the <user> is simply the username:

OpenAI compatible Model Configuration

Because the authentication proxy uses the OpenAI API, a model needs to be configured as an OpenAI or InspireLLM model.

Creating an OpenAI compatible Model

Log in to the Mindbreeze InSpire Management Center.
In the left-hand navigation pane, select “Insight Services” -> “RAG” -> “LLMs”.
Click “Add”. Select OpenAI
Enter the following details:
- Model: <model-name> (should match the models.model in authentication proxy configuration)
- Use custom LLM URL: On
- URL: https://<endpoint>.<clusterExternalDomain>/api/llm/service-account-mb-inspire-llm-client/<models.model>/v1 (should match the configuration from the LLM Proxy, not Authentication Proxy).

Give model a name.
To test the inference engine, navigate to “Test” section and enter a prompt. If you get an answer, then an LLM has been successfully configured.
Save the model.

Creating an InSpire LLM Model

Log in to the Mindbreeze InSpire Management Center.
In the left-hand navigation pane, select “Insight Services” -> “RAG” -> “LLMs”.
Click “Add”. Select InSpire LLM
Enter the following details:
- URL: https://<endpoint>.<clusterExternalDomain>/api/llm/service-account-mb-inspire-llm-client/<models.model> (should match the configuration from the LLM Proxy, not Authentication Proxy) Make sure the <models.model> matches the model you want to use.

Give model a name.
To test the inference engine, navigate to “Test” section and enter a prompt. If you get an answer, then an LLM has been successfully configured.
Save the model.

Using Huggingface TGI API (Legacy)

You can use Huggingface TGI APIs directly without authentication. To do this, you need to add the supportLegacyUnauthenticatedTGI flag in the llm-proxy configuration and set it to true. For example:

llm-proxy:

enabled: true

...

supportLegacyUnauthenticatedTGI: true

models:

- model: default-text-llm-vllm

charts:

- default-text-llm-vllm

- model: default-text-llm-tgi

charts:

- default-text-llm-tgi

apiProxy:

charts:

api-llm-proxy

...

This way you are still using the Huggingface TGI API, but without authentication.

Configuring TGI Model in RAG

Log in to the Mindbreeze InSpire Management Center.
In the left-hand navigation pane, select “Insight Services” -> “RAG” -> “LLMs”.
Click “Add”. Select InSpire LLM (TGI)
Enter the following details:
- URL: (should match the configuration from the LLM Proxy, not Authentication Proxy)

Give model a name.
To test the inference engine, navigate to “Test” section and enter a prompt. If you get an answer, then an LLM has been successfully configured.
Save the model.

Note: This configuration would not have authentication enabled.

Disabling Authentication

By default, authentication is enabled within the api-llm-proxy. However, in environments where authentication is not required, it can be deactivated by setting the auth.enabled field to “false”.

When this setting is applied, Keycloak integration is bypassed, and authentication is skipped.

api-llm-proxy:

...

auth:

enabled: false

Use Cases

The mindbreeze-api-llm-proxy is the control point between clients and the deployed models. Because all traffic to the models passes through it, it is where cross-cutting governance is applied – without modifying the models themselves. Built on LiteLLM, the proxy exposes a single OpenAI-compatible endpoint and lets administrators decide, per model, who may call it, how often, and at what cost.

The sections below describe the main use cases the proxy

Role-based Access Control

It is possible to restrict a model only for users that have a specific role. In order to do this, you can configure the required_roles parameter in the authentication proxy configuration. For example:

api-llm-proxy:

...

models:

- model: default-text-llm-vllm

model_id: unsloth/Qwen3-32B-unsloth-bnb-4bit

chart: default-text-llm-vllm

requests_per_minute: 100

- model: default-text-llm-tgi

model_id: meta-llama/Llama-3.1-8B-Instruct

chart: default-text-llm-tgi

requests_per_minute: 100

required_roles: ["Meta Llama Model Access"]

Only users with the Meta Llama Model Access role can use the TGI model. By default, if required_roles are not set, user requires to have InSpire LLM Services User role. required_roles parameter is an array, so several roles might be defined.

Model Discovery & Registry

The mindbreeze-api-llm-proxy also acts as a central registry for the models and agents exposed across the enterprise. Every model that should be reachable is declared once, under the proxy’s models list, giving administrator’s a single place to govern what is deployed – rather than tracking scattered, per-team endpoints.

Model discovery builds on the Hugging Face Hub rather than reinventing it: models are referenced by their canonical Hugging Face model_id (for example, google/gemma-4-31B-it), which pins both the model and its revision and lets you rely on Hugging Face’s existing catalog, metadata, and version history. The proxy entry then maps that model to the chart that serves it and to the access and usage policies that apply.

api-llm-proxy:

models:

- model: default-text-llm # Exposed model name

model_id: meta-llama/Llama-3.1-8B-Instruct # discovered & version-pinned via Hugging Face Hub

chart: default-text-llm-vllm

required_roles: ["Meta Llama Model Access"]

requests_per_minute: 100

Because each entry couples the model identity (model_id), the serving deployment (chart), and its governance (required roles, rate limits, pricing), the models list doubles as the authoritative catalog of what is available and how it is governed.

While any Hugging Face model can in principle be referenced, we maintain a curated list of supported models, that have been validated against a rigid, standardized test suite covering deployment, inference, and runtime compatibility across the supported engines (TGI, vLLM, SGLang). Choosing a model from this list ensures predictable behavior in Mindbreeze Inspire and a known-good configuration. The full list of supported and tested models is available here [Supported/Tested Models and Configurations].

Discovering exposed models at runtime. Consumers do not need access to the values.yaml to find out which models they can use. The proxy exposes the OpenAI-compatible GET /v1/models endpoint, which returns the catalog of currently exposed models. Because the request passes through the same authentication and role checks as inference calls, the response is scoped to the caller: each user (or team/organization, when impersonation is enabled) sees only the models their roles permit – a model gated behind required_roles is omitted from the listing for anyone lacking that role. This lets clients discover the models available to a specific user dynamically, without hardcoding model IDs.

curl https://<proxy-host>/api/llm/v1/models -H “Authorization: Bearer <token>”

Rate-limiting & Cost Management

Rate limiting and cost management are two sides of the same control: the first caps how much a model can be used, the second measures what that usage costs. Both are configured per model on the mindbreeze-api-llm-proxy, so different models can carry different quotas and prices.

Rate-limiting

Use requests_per_minute and tokens_per_minute to limit usage:

api-llm-proxy:

models:

- model: default-text-llm

model_id: meta-llama/Llama-3.1-8B-Instruct

chart: default-text-llm

requests_per_minute: 100

tokens_per_minute: 10000

- model: default-multimodal-llm

model_id: google/gemma-4-31B-it

chart: default-multimodal-llm

requests_per_minute: 1000

Here, model default-text-llm is limited to 100 requests and 10,000 tokens per minute, while default-multimodal-llm can handle 1,000 requests per minute.

The two limits are independent and evaluated together – whichever is reached first applies. requests_per_minute guards against high call volume (many small requests), while tokens_per_minute guards against high load (fewer requests, but each with large prompts or long generations), which is often the more accurate proxy for actual GPU pressure. Setting one leaves the other dimension unbounded, so for capacity-sensitive models it is usually worth defining both. When a limit is exceeded, further requests to that model are rejected (HTTP 429) until the window resets; other models are unaffected, since limits are scoped per model rather than shared across the proxy. If a limit is omitted, that dimension is treated as unlimited for the model. Beyond protecting GPU capacity and keeping latency predictable under load, these quotas are also the first line of cost control — they cap the maximum spend a model can generate in any given minute.

Cost Management

Where quotas cap usage, per-token pricing values it. LiteLLM tracks token usage on every request, and administrators can assign a price per token to each model, letting the proxy convert raw token counts into a monetary spend figure per request, model, and caller.

Pricing uses LiteLLM’s standard cost parameters.

api-llm-proxy:

models:

- model: default-multimodal-llm

model_id: google/gemma-4-31B-it

chart: default-multimodal-llm

model_info:

input_cost_per_token: 0.0000002 # price charged per input token

output_cost_per_token: 0.0000006 # price charged per output token

Forwarding cost data (webhooks). The spend calculated per request can be pushed to your own billing, or monitoring systems through the webhook integration described in Header-based Impersonation & Webhook Integration . When both impersonation and a webhook are enabled, LiteLLM includes the resolved organization_id, team_id, and user_id alongside token usage and computed spend in each webhook payload – so cost is attributed to the correct organization, team, or user without any additional parsing on the receiving side. Together, the registry, quotas, per-token pricing, and webhooks form a closed loop: models are discovered and cataloged, quotas cap their usage, pricing values that usage, and webhooks deliver the resulting cost data to the systems that need it.

Guardrails: Default & Custom Enforcement

Guardrails are defined declaratively in the Helm chart's `values.yaml`, keeping policy version-controlled and consistent across every environment.

Default guardrails - set `default_on: true` to enforce a check on every request automatically, no client opt-in needed. This is how non-negotiable baseline policy (e.g., blocking SSNs/credit cards) gets applied organization-wide:

config:

guardrails:

- guardrail_name: "baseline-pii-block"

litellm_params:

guardrail: litellm_content_filter

mode: "pre_call"

default_on: true

pii_entities_config:

US_SSN: "BLOCK"

Custom guardrails - anything without `default_on` is available but not auto-applied; the calling app opts in per-request, or teams write fully bespoke sandboxed logic for organization-specific rules.

Event hooks - each guardrail runs `pre_call` (before the LLM call, on input), `post_call` (after the call, on input and output), or `during_call` (parallel to the call, blocking the response until the check completes), and a single guardrail can run in multiple modes at once. This lets latency-sensitive paths use `pre_call` blocking checks while lower-risk paths log via `post_call` without holding up the response.

Built-in vs. third-party - the built-in `litellm_content_filter` handles baseline PII, keyword, and content-category filtering with no external API dependency. For ML-based detection like jailbreak or toxicity scoring, third-party guardrails (Presidio, Aporia, Lakera, Bedrock, IBM, etc.) or a generic guardrail API plug into the same config, using the same hooks and `default_on` behavior.

Scope beyond chat - the same config pattern extends to tool-call permissions (allow/deny rules on `tool_calls`/MCP tools), MCP pre-call validation, and realtime/voice sessions, so guardrail policy isn't limited to standard chat completions.

Header-based Impersonation & Webhook Integration

This guide explains header-based user impersonation to the LiteLLM proxy. An upstream gateway can forward identity context – organization, team, and user – via HTTP request headers. The proxy validates and passes these values to LiteLLM, making them available for routing, logging, and webhooks.

Refer to the link below for the full configuration spectrum:

mindbreeze-api-llm-proxy: Section 12.1.2.4 .

Configuring impersonation headers

Impersonation is configured under auth.impersonation in values.yaml. It requires auth.enabled: true to take effect.

api-llm-proxy:

auth:

enabled: true

impersonation:

enabled: true

headers:

organization:

allowedPattern: null

team:

allowedPattern: null

user:

allowedPattern: null

Parameters

Parameter	Explanation
name	The name of the HTTP header the proxy reads from each incoming request. The upstream system must set this header before the request reaches the proxy.
allowedPattern	Defines an optional regular expression that the header values must fully match. If the value does not match, it is silently dropped and the corresponding identity field is left unset. Set to null to accept any value without validation.

Examples

Accept only UUIDs for organization IDs:

organization:

allowedPattern: ‘^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$’

Accept only lowercase alphanumeric team IDs:

team:

allowedPattern: ‘^[a-z0-9-]+$’

Accept any user value (no validation):

user:

allowedPattern: null

Registering a webhook

Webhooks are registered through LiteLLM’s callback_settings configuration. Because the .config namespace is a transparent pass-through to LiteLLM’s config.yaml, all LiteLLM configuration options are available directly under .config in values.yaml.

To register a webhook, add it under config.callback_settings and reference the callback name in config.litellm_settings.callbacks:

api-llm-proxy:

config:

litellm_settings:

callbacks:

- generic

callback_settings:

generic:

callback_type: “generic_api”

url: "https://your-webhook-endpoint.example.com/events"

headers: {}

Important: The setting headers: {} is required on every webhook entry. Omitting it will cause the webhook registration to fail.

Using impersonation data in a webhook

When impersonation is enabled, the resolved identity fields (organization_id, team_id, user_id) are attached to each LiteLLM request context. LiteLLM includes this metadata in the payload it sends to registered webhooks, allowing the receiving system to attribute usage to the correct organization, team, or user without re-parsing the original request headers.

No additional configuration is needed – enabling both impersonation and a webhook is sufficient for the identity context to appear in webhooks payloads.

api-llm-proxy:

auth:

enabled: true

impersonation:

enabled: true

headers:

organization:

allowedPattern: null

team:

allowedPattern: null

user:

allowedPattern: null

config:

litellm_settings:

callbacks:

- generic

callback_settings:

generic:

callback-type: “generic_api”

url: "https://your-webhook-endpoint.example.com/events"

headers: {}

Network Policies

This guide explains how to configure and manage the Kubernetes Network Policies for the multi-tier InSpire LLM deployment. These policies enforce strict network isolation to secure the cluster environment, allowing traffic only along explicitly defined paths.

Refer to the links below for the full configuration spectrum:

mindbreeze-llm-proxy: Section 12.1.3.11
mindbreeze-api-llm-proxy: Section 12.1.2.7
mindbreeze-llm-inference: Section 12.1.1.11

Architecture & Traffic Flow

The deployment consists of three distinct tiers

Edge Tier (ingress-proxy): Handles incoming traffic from OpenShift routers or external sources.
Middleware Tier (api-proxy): Processes requests, handles authentication (e.g., via Keycloak), and routes them to the backend.
Backend Tier (inference-engine): Executes the LLM inference.

Standard Secure Flow: External Traffic -> ingress-proxy -> api-proxy -> inference-engine.

Global Configurations

These configurations apply to all three Helm charts:

mindbreeze-llm-proxy
mindbreeze-api-llm-proxy
mindbreeze-llm-inference

Enabling Network Policies

To enable network isolation, you must explicitly enable it in your values.yaml for each chart.

networkPolicy:

enabled: true

The “Allow All” Override

For troubleshooting or development, you can temporarily bypass all isolation. When a network policy is enabled for a chart, the allowAll flag is set to true per default. To enforce strict network policies you need to disable it.

Warning: Setting allowAll: true completely disables network isolation for the pod, allowing all inbound and outbound traffic. A warning will be displayed in the Helm release notes if this is active.

networkPolicy:

enabled: true

allowAll: true # NOT RECOMMENDED FOR PRODUCTION

DNS Resolution

Pods typically need to resolve internal Kubernetes services or external domains. Enable egress to the cluster's DNS servers (Port 53/5353):

networkPolicy:

enabled: true

allowAll: false

dns:

enabled: true

allowExternal: false # Set to true if using DNS outside the cluster namespace

Common Configuration Use Cases

Below are the configurations required for the most common deployment scenarios.

Scenario A: Enabling the Standard Multi-Tier Flow

To allow the intended traffic flow from the Edge through the Middleware to the Backend, configure the values.yaml of each component as follows:

Ingress Proxy (mindbreeze-llm-proxy)

Allow outbound traffic to the API proxy:

networkPolicy:

enabled: true

allowAll: false

dns:

enabled: true

apiProxy:

enabled: true

API Proxy (mindbreeze-api-llm-proxy)

Allow inbound traffic from the Ingress proxy, and outbound traffic to the Inference Engine:

networkPolicy:

enabled: true

allowAll: false

dns:

enabled: true

ingressProxy:

enabled: true

inferenceEngine:

enabled: true

ports:

- 8080 # Default inference port

Inference Engine (mindbreeze-llm-inference)

Allow inbound traffic from the API proxy:

networkPolicy:

enabled: true

allowAll: false

dns:

enabled: true

apiProxy:

enabled: true

ports:

- 8080

Scenario B: Enabling Internet Access for Model Downloads

By default, the inference-engine is completely isolated and cannot reach the internet to download models from repositories like HuggingFace.

Configure the mindbreeze-llm-inference chart to allow external egress. The policy is automatically configured to block private internal subnets (10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12) while allowing public internet access.

networkPolicy:

enabled: true

allowAll: false

dns:

enabled: true

internetAccess:

enabled: true

onlyHTTPS: true # Recommended: Restricts traffic to port 443 only

# ports: [] # Use this to define additional required external ports

Scenario C: Connecting API Proxy to Keycloak

If your api-proxy requires authentication via Keycloak, it needs egress permissions to reach the Keycloak instance.

Configure the mindbreeze-api-llm-proxy chart.

Note: You must provide the exact address block (IP block or Namespace/Pod selectors) for Keycloak, or the deployment will fail.

networkPolicy:

enabled: true

allowAll: false

dns:

enabled: true

keycloak:

enabled: true

port: 80 # Or 443, depending on your Keycloak setup

address:

- ipBlock:

cidr: <keycloak-ip> # Example IP for Keycloak

# OR using namespace selectors if Keycloak is inside the cluster:

# - namespaceSelector:

# matchLabels:

# kubernetes.io/metadata.name: keycloak-namespace

Scenario D: Allowing External Traffic into the Ingress Proxy

To allow users or other systems to reach your edge tier, configure the mindbreeze-llm-proxy chart.

For OpenShift Environments:

networkPolicy:

enabled: true

allowAll: false

ingressController:

enabled: true # Allows traffic from OpenShift default ingress routers

hostNetwork:

enabled: true # Allows traffic from the OpenShift host network

For Standard External Access (Raw IP):

networkPolicy:

enabled: true

allowAll: false

externalAccess:

enabled: true

allowedCIDRs:

- 0.0.0.0/0 # Allows the entire internet (use specific CIDRs for restricted access)

Advanced Rules

If you need to define connections to external databases, logging systems, or custom integrations not covered by the default flags, you can inject raw Kubernetes NetworkPolicy rules using the additionalIngressRules and additionalEgressRules blocks in any of the three charts.

Example: Allowing an external monitoring tool to scrape metrics on port 9090:

networkPolicy:

enabled: true

allowAll: false

additionalIngressRules:

- from:

- ipBlock:

cidr: <monitoring-server-IP> # IP of the monitoring server

ports:

- protocol: TCP

port: 9090

Scenarios

Deploying Llama-3.1-8B-instruct with Hugging Face TGI using ArgoCD and exposing it externally

To deploy a model and expose it externally, we need to create two deployments, one for mindbreeze-llm-inference, and the other for mindbreeze-llm-proxy.

Clone the repository

To deploy the system on ArgoCD, you need to first clone the repository. For example, to deploy on mb-main-development, clone the repository.

Add dependencies

First step is to add two new dependencies to the Chart.yaml.

apiVersion: v2

description: MB Main Deployment

version: 0.0.1

dependencies:

………………………………………

- name: mindbreeze-llm-proxy

version: <latest-helm-chart-version>

repository: alias:<alias>

alias: llm-proxy

- name: mindbreeze-llm-inference

version: <latest-helm-chart-version>

repository: alias:<alias>

alias: default-text-llm

This configuration states that we need two additional dependencies (in this case deployments), one for the mindbreeze-llm-proxy called llm-proxy, and the other one for the mindbreeze-llm-inference, called default-text-llm.

Apply the deployment configuration

After you have specified the dependencies, you can configure them as you want. You need to write the configuration in the values.yaml file. In addition, configuration for each dependency should be namespaced by its alias.

Now append the following configuration to values.yaml.

Model Deployment

default-text-llm: # alias

enabled: true

instanceCount: 2

image:

repository: "<repository>"

baseName: "<baseName>"

registryPullSecrets: <registry-pull-secrets>

runtime: “tgi”

model:

id: "meta-llama/Llama-3.1-8B-Instruct”

hub:

tokenSecret: hf-token-secret

gpu:

enabled: true

profile: "nvidia.com/mig-1g.70gb" # Make sure profile is available

resourcesMap:

storage: 150Gi

requests:

cpu: 5

memory: 20Gi

limits:

cpu: 10

memory: 50Gi

storageClassName: <storage-class-name>

LLM Proxy

llm-proxy: # alias

instanceCount: 2

image:

repository: "<repository>"

baseName: "<baseName>"

registryPullSecrets: <registry-pull-secrets>

clusterExternalDomain: apps.openshift.com

ingressTLSSecret: custom-certs-default

endpoint: llms

models:

- model: meta-llama-3.1-8b-instruct

charts:

- default-text-llm

limits:

connectionsPerUser: 4

connectionsGlobal: 20

Push the changes

Push the changes to the git repository. After this, navigate to the ArgoCD main page and find the application: mb-main-development. Press a refresh button.

Notes

The deployment may take a couple of minutes, because the model may have to be downloaded.

Access URL

If the deployments were successful, you can now access the meta-llama model externally using the following URL:

https://llms.apps.openshift.com/api/llm/some-user/meta-llama-3.1-8b-instruct/<api>

Setting a Specific GPU Device

If the cluster you are deploying to supports multiple types of NVIDIA GPUs, you can specify which GPU device to use in the deployment configuration file for the LLM.

To set a specific GPU device, configure the gpu.product field under the deployment settings.

Example:

default-text-llm:

enabled: true

...

gpu:

enabled: true

profile: "nvidia.com/gpu"

product: “NVIDIA-H100-NVL”

In the above example, the model will be deployed on a NVIDIA-H100-NVL GPU if it’s available.

Note: The product value must exactly match one the available GPU models on the cluster. If the value is incorrect or the specified GPU is unavailable, the pod may remain in a pending state.

Deploying a Model on CPU

With the InSpire LLM 25.6 release (or later), you can now deploy any supported model on a CPU by swapping out the GPU-oriented runtime for the new “vllm-cpu” runtime. The steps are identical to a GPU deployment, with only the runtime field changing.

Below is a minimal example illustrating the change:

default-text-llm:

…

runtime: “vllm-cpu”

…

Notes on CPU Deployments

Performance: CPU inference will generally be slower than GPU.
Monitoring: Keep an eye on CPU utilization and memory usage – some models can be quite large.

Deploying and Using Speech-to-Text (STT) Models

A dedicated runtime framework has been introduced for handling Speech-to-Text (STT) tasks. This runtime, named “audio-default-engine”, supports the following capabilities:

Transcription of long audio files
Inclusion of timestamp data (segment-level timing)

Deploying the STT Model

To deploy an STT model, make sure you configuration includes the following settings:

[Chart.yaml]

...

dependencies:

- name: mindbreeze-llm-proxy

version: <latest-helm-chart-version>

repository: alias:<alias>

alias: llm-proxy

- name: mindbreeze-llm-inference

version: <latest-helm-chart-version>

repository: alias:<alias>

alias: default-audio-llm

- name: mindbreeze-api-llm-proxy

version: <latest-helm-chart-version>

repository: alias:<alias>

alias: api-llm-proxy

[values.yaml]

default-audio-llm:

runtime: "audio-default-engine"

audioTranscriptions:

models:

id: "large-v3-turbo"

...

llm-proxy:

instanceCount: 2

ingressTimeout: 120

apiProxy:

charts:

api-llm-proxy

api-llm-proxy:

models:

- model: large-v3-turbo

model_id: large-v3-turbo

chart: default-audio-llm

Supported Model IDs:

Model ID	Description
large-v3	Corresponds to Systran/faster-whisper-large-v3
large-v3-turbo	Corresponds to rtlingo/mobiuslabsgmbh-faster-whisper-large-v3-turbo
<model-id>	Other model ids, supported by the speaches-ai inference engine. To list supported model, run curl localhost:8080/v1/registry inside a model pod.

Making Requests (Speaches via Proxy)

To transcribe audio using the “audio-default-engine” runtime through the proxy, use the following curl command:

curl https://<endpoint>.<clusterExternalDomain>/api/llm/mindbreeze-user/default-audio-llm/v1/audio/transcriptions \

-H "Content-Type: multipart/form-data" \

-F file=@/path/to/audio.mp3 \

-F prompt="Transcribe this audio." \

-F language=en \

-F model=”<model-id>” # Either large-v3 or large-v3-turbo

-F stream=True \

-F response_format=["text", "json", "verbose_json", "srt", "vtt"] # Choose one

Timestamps: To include segment-level timestamps, use response_format values verbose_json, srt, or vtt.
Prompt: Providing a prompt can improve transcription accuracy.
Request format: Only multipart/form-data is supported. JSON requests are not supported.

Deploying and Using Multimodal Models

Multimodal models can be deployed using either the Hugging Face TGI or vLLM backend runtimes. The configuration format is similar to other model deployments:

default-multimodal-llm:

enabled: true

image:

repository: "<repository>"

baseName: "<baseName>"

runtime: “<runtime>”# ‘tgi’ or ‘vllm’

model:

id: "<multimodal model>”

maxTotalTokens: <max-total-tokens>

Example: Deploying Qwen/Qwen2.5-VL-72B-Instruct-AWQ on NVIDIA H100 NVL Using vLLM Runtime

Qwen/Qwen2.5-VL-72B-Instruct-AWQ is a quantized vision-language model that supports both image and video-to-text inference.

To deploy this model, use the following configuration:

[Chart.yaml]

…

dependencies:

- name: mindbreeze-llm-proxy

version: <latest-helm-chart-version>

repository: alias:<alias>

alias: llm-proxy

- name: mindbreeze-llm-inference

version: <latest-helm-chart-version>

repository: alias:<alias>

alias: default-multimodal-llm

[values.yaml]

default-multimodal-llm:

enabled: true

runtime: “vllm”

model:

id: “Qwen/Qwen2.5-VL-72B-Instruct-AWQ”

maxTotalTokens: 32000

quantization: “awq_marlin”

...

llm-proxy:

models:

- model: default-multimodal-llm

charts:

- default-multimodal-llm

...

Making Requests via Proxy

Both Hugging Face TGI and vLLM deployments support the following request formats for multimodal inputs.

Sending an Image

To send an image, make the following request:

curl https://<endpoint>.<clusterExternalDomain>/api/llm/<user>/default-audio-llm/v1/chat/completions \

-H "Content-Type: application/json" \

-d '

{

"message": [

{

"role": "user",

"content": [

{

"type": "text",

"text": "What is depicted in this image?"

{

"type": "image_url",

"image_url": {

"url": "data:image/<image-format>;base64,<base64-encoded-image>"

}

]

}

]

Sending a Video

To send a video, make the following request:

curl https://<endpoint>.<clusterExternalDomain>/api/llm/<user>/default-audio-llm/v1/chat/completions \

-H "Content-Type: application/json" \

-d '

{

"message": [

{

"role": "user",

"content": [

{

"type": "text",

"text": "What is depicted in this video?"

{

"type": "video_url",

"video_url": {

"url": "data:video/<video-format>;base64,<base64-encoded-video>"

}

]

}

]

Note:

This request format works for both TGI and vLLM backend.

Deploying google/gemma-3-27b-it

To successfully deploy the google/gemma-3-27b-it model family using the vLLM engine, specific hardware and configuration requirements must be met.

Requirements

Data type: The model must be run with dtype: bfloat16.

GPU: Deployment requires a high-memory GPU such as the NVIDIA H100 (e.g., H100 NVL).

Model Access: The model is gated on Hugging Face. Ensure your Hugging Face token has been granted access to the google/gemma-3-27b-it model.

Example Configuration

gemma-3-27b-it:

...

runtime: "vllm"

model:

id: "google/gemma-3-27b-it"

dtype: "bfloat16"

...

gpu:

enabled: true

profile: "nvidia.com/gpu"

product: "NVIDIA-H100-NVL"

resourcesMap:

storage: 150Gi

requests:

cpu: 5

memory: 40Gi

limits:

cpu: 10

memory: 100Gi

It is recommended to deploy the model with the inference engine vLLM or SGLang. Running with Huggingface TGI introduces some instability.

Deploying Open Source GPT Models

Starting from InSpire LLM version 25.7, you can deploy the openai/gpt-oss-20b and openai/gpt-oss-120b models.

Requirements

GPU Hardware: A high-memory GPU such as the NVIDIA H100 (e.g., H100 NVL).
Model Access: GPT models are not gated, so no special access request is required. However, a valid Hugging Face access token secret is still required.

Example Configuration

default-text-llm:

runtime: "vllm"

model:

id: "openai/gpt-oss-120b" # or "openai/gpt-oss-20b"

maxTotalTokens: 90000

gpu:

enabled: true

profile: "nvidia.com/gpu"

product: "NVIDIA-H100-NVL"

resourcesMap:

storage: 150Gi

requests:

cpu: 10

memory: 60Gi

limits:

cpu: 40

memory: 90Gi

Deployment and Usage of the DeepSeek-OCR Model

Starting from InSpire LLM version 25.9, you can deploy the deepseek-ai/DeepSeek-OCR model using either the vLLM or SGLang inference engines.

Requirements

GPU Hardware: Recommended, but not necessary. If deploying on a CPU, make sure to use vLLM inference engine for CPU.
Model Access: deepseek-ai/DeepSeek-OCR model is not gated, so no special access request is required. However, a valid Hugging Face access token secret is still required.

Deployment

vLLM

default-ocr:

runtime: "vllm"

model:

id: "deepseek-ai/DeepSeek-OCR"

maxModelLen: 8192

maxNumBatchedTokens: 8192

logitsProcessors: vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor

noEnablePrefixCaching: true

mmProcessorCacheGb: 0

SGLang

default-ocr:

runtime: "sglang"

model:

id: "deepseek-ai/DeepSeek-OCR"

maxTotalTokens: 8192

maxPrefillTokens: 8192

Usage

To effectively use the deepseek-ai/DeepSeek-OCR feature, you must interact carefully with the provided API. While the deployed model supports the v1/chat/completions endpoint, its usage and prompt structure must follow a specific format.

Before sending a document to the model, convert each page into an image and Base64-encode it. Once the images are encoded, you can send requests using the payload format shown below. For instance, this how a JSON payload should look like:

{

"model": "default-ocr",

"temperature": 0.0,

"extra_body": { # Only with the vLLM deployment

"skip_special_tokens": false,

"vllm_xargs": {

"ngram_size": 30,

"window_size": 90

}

"messages": [

{

"role": "user",

"content": [

{

"type": "text",

"text": "<|grounding|>Convert the document to html."

{

"type": "image_url",

"image_url": {

“url”: “<base_64_encoded_image>”

}

]

}

]

}

Deployment of Mistral AI Models

Requirements

GPU Hardware
Model Access
The supported Mistral AI models are not gated, so no special access request is required. However, a valid Hugging Face access token is still required.

mistralai/Mistral-Small-3.2-24B-Instruct-2506

default-text-llm:

runtime: "vllm"

model:

id: "mistralai/Mistral-Small-3.2-24B-Instruct-2506"

maxModelLen: 24575

maxNumBatchedTokens: 24576

loadFormat: mistral

tokenizerMode: mistral

configFormat: mistral

...

RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8

default-text-llm:

runtime: "vllm"

model:

id: "RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8"

maxModelLen: 24575

maxNumBatchedTokens: 24576

loadFormat: mistral

tokenizerMode: mistral

configFormat: mistral

enableAutoToolChoice: true

...

Configuring a LoadBalancer Service for HTTPS Access

This setup is useful for production scenarios where public or private IP access to services is required without relying on an ingress controller or domain name resolution. By using a TLS secret, HTTPS encryption is handled by NGINX, and traffic is securely routed to the appropriate backend.

Example Configuration

llm-proxy:

…

ingress:

enabled: false

nginx:

useHTTPS: true

TLSSecret: router-certs-default

service:

type: LoadBalancer

loadBalancerIP: <loadbalancer-ip>

Key Parameters	Description
ingress.enabled: false	Disables ingress and opting to expose the service directly via LoadBalancer.
nginx.useHTTPS: true	Enables HTTPS support in NGINX.
nginx.TLSSecret: router-certs-default	Specifies the Kubernetes TLS secret containing the certificate and private key for HTTPS.
service.type: LoadBalancer	Configures the service to be exposed using a LoadBalancer.
service.loadBalancerIP: <ip-address>	Assigns a specific external IP address to the service.

Usage

Once the configuration is applied and the service is deployed, it will be accessible over HTTPS via the specified LoadBalancer IP. By default, NGINX will serve traffic on port 8443.

You can verify access using curl:

curl https://<ip-address>:8443/<api>

Replace <api> with the appropriate endpoint path for your application.

Hint: If your TLS certificate is self-signed or issued by a private CA, you may need to use the command curl -k to bypass certificate validation during testing.

Consuming the API

There are two ways to consume the API: Legacy Specification and the OpenAI API Compatible Standard.

Legacy Specification

In the Legacy Specification, the model is specified directly in the URL endpoint. After successful deployment and exposure of the ingress URL, the legacy API uses the following URL template:

https://<ingress>/api/llm/<keycloak-user>/<model-name>/<api>

Where <api> can be one of the following:

v1/chat/completions
v1/audio/transcriptions

Note: This is the legacy version of the API. It is recommended to use the latest OpenAI API compatible standard, available with the InSpire LLM 25.6 release.

OpenAI API Compatible Standard

With the InSpire LLM 25.6 release, it is recommended to use the OpenAI API Compatible Standard for accessing the API.

After successful deployment and exposure of the ingress URL, use the following endpoint:

https://<ingress>/api/llm/<api>

Where <api> can be one of the following:

v1/chat/completions
v1/audio/transcriptions
v1/models

In this version, the model name is specified within the JSON payload or as part of a multipart/form-data request, rather than in the URL.

Deploying Large Models on Multiple GPUs

When a model’s parameter count exceeds the memory capacity of a single GPU, you must distribute the workload across multiple devices.

vLLM

vLLM utilizes Tensor Parallelism (TP) and Pipeline Parallelism to manage memory constraints. Use the following logic to determine your strategy:

Single GPU: If the model fits on one device, distributed inference is unnecessary.
Single-Node, Multi-GPU: Use Tensor Parallelism to split weights across GPUs on the same machine.
- Example: For a node with 4 GPUs, set tensorParallelSize: 4.
Multi-Node, Multi-GPU: If the model exceeds the memory of a single node, combine Tensor Parallelism (within the node) and Pipeline Parallelism (across nodes).
- Example: Using 2 nodes with 8 GPUs each, set tensorParallelSize: 8 and pipelineParallelSize: 2.

Configuration Example

default-text-llm:

runtime: "vllm"

model:

id: "MiniMaxAI/MiniMax-M2.5"

tensorParallelSize: 8

pipelineParallelSize: 2

SGLang

SGLang offers flexible parallelism through its runtime arguments. Note that SGLang focuses heavily on Data Parallelism (DP) for throughput and Tensor Parallelism (TP) for memory.

TP (Tensor Parallelism): Used to shard the model weights.
- Example: For a node with 4 GPUs, set tp: 2.
DP (Data Parallelism): Used to run multiple independent replicas of the model to handle higher request volumes.
- Example: To enable multi-GPU data parallelism, add dp: 2. It can also be used together with tensor parallelism. Configuring dp: 2 and tp: 2 requires a total of 4 GPUs.

Configuration Example

default-text-llm:

runtime: "sglang"

model:

id: "MiniMaxAI/MiniMax-M2.5"

tp: 8

dp: 2

If your InSpire LLM version does not support these arguments, you can specify them in the args field:

default-text-llm:

runtime: "sglang"

model:

id: "MiniMaxAI/MiniMax-M2.5"

args:

- "--tp 8"

- "--dp 2"

Hugging Face TGI

Hugging Face TGI simplifies the process using the sharding concept. It automatically handles the underlying NCCL communication required for multi-GPU setups.

numShard: Set this to the total number of GPUs you wish to utilize.
sharded: This boolean must be enabled to allow the model to be partitioned across the specified shards.

Configuration Example

default-text-llm:

runtime: "tgi"

model:

id: "MiniMaxAI/MiniMax-M2.5"

numShard: 8

sharded: true

Deploying Gemma 4 Models

Starting from InSpire LLM version 26.3, you can deploy the Gemma 4 models.

Requirements

GPU Hardware: A high-memory GPU such as the NVIDIA Ampere, Hopper or other generations (e.g., H100 NVL).
Model Access: Gemma 4 models are not gated, so no special access request is required. However, a valid Hugging Face access token secret is still required.
Inference Engine: Deploying Gemma 4 models is only possible with vLLM engine.

Configuration Example

gemma-4-31b-it:

instanceCount: 1

registryPullSecrets: <registry-pull-secrets>

runtime: "vllm"

model:

id: "google/gemma-4-31B-it"

maxModelLen: 90000

maxNumBatchedTokens: 16384

hub:

tokenSecret: hf-token-secret

gpu:

enabled: true

profile: "nvidia.com/gpu"

product: "NVIDIA-H100-NVL"

resourcesMap:

storage: 100Gi

requests:

cpu: 5

memory: 40Gi

limits:

cpu: 20

memory: 90Gi

readinessProbe:

initialDelaySeconds: 60

periodSeconds: 10

failureThreshold: 200

livenessProbe:

initialDelaySeconds: 60

periodSeconds: 30

failureThreshold: 70

Using the Deployed Model in the Inspire RAG Service

Once your model and proxy are deployed, you can register it as a new LLM in the Inspire RAG service.

To add a new model:

Log in to your Inspire environment.
In the Management Center, select RAG Service.
Open the LLMs section.
Click “Add”.

Choose “InSpire LLM”
In the URL field, paste your deployment’s base URL. For example:

https://<endpoint>.<clusterExternalDomain>/api/llm/mindbreeze-user/default-text-llm

Under the General tab, enter a model name.
Switch to the Test section and verify the model responds as expected.
Click Save to complete model registration.

After these steps, your new model will be available for use in a RAG pipeline.

Explanation of Quantization

To deploy a model with quantization in Hugging Face TGI, the model must already be quantized beforehand. Specifying the quantization argument alone does not perform quantization – it simply tells TGI or vLLM how to load a model that’s already been quantized.

For example, deploying models like meta-llama/Llama-3.1-8B-Instruct or deepseek-ai/DeepSeek-R1-Distill-Qwen-14B with the quantization argument will not work unless those models have been pre-quantized.

Quantized models typically include a suffix indicating the quantization method used. For instance, the model TheBloke/deepseek-llm-67b-base-AWQ has been quantized using the AWQ method.

When deploying a quantized model with TGI or vLLM, make sure to also specify the appropriate quantization value in your values.yaml:

model:

id: " TheBloke/deepseek-llm-67b-base-AWQ”

quantization: “awq”

Important: Keep in mind that quantization may cause instability. Use it only with models that are already pre-quantized using a known quantization method.

Multi-Model Pod Deployments

The mindbreeze-llm-inference chart now supports deploying multiple AI models within a single Kubernetes Pod. This is achieved by running multiple inference engine containers side-by-side in the same Pod, each exposing a different port.

While this feature is supported for specific edge cases, we strongly recommend deploying a single model per Pod (via separate Helm releases) for production environments. Please read the section "" carefully before using this feature.

Configuration Inheritance

When configuring multiple models, the Helm chart utilizes an Inheritance and Override principle.

In your values.yaml, you have global configurations at the root level (e.g., image, gpu, env). You also have a new models: array.

Inheritance: If a configuration is missing inside a specific item in the models: array, the container will automatically inherit the global setting.
Override: If you specify a configuration inside an item in the models: array, it will override the global default for that specific container only.

Configuring Multiple Models

To deploy multiple models, define them as a list under the models: key in the mindbreeze-llm-inference values.yaml file.

Port Allocation

Ports are assigned dynamically based on the model's index in the list:

Model 0 (First in list): Container Port 8080 / Service Port 80
Model 1 (Second in list): Container Port 8081 / Service Port 81
Model 2 (Third in list): Container Port 8082 / Service Port 82

Example Configuration (mindbreeze-llm-inference)

# GLOBAL DEFAULTS

runtime: "vllm"

gpu:

enabled: true

resourcesMap:

requests:

cpu: 2

memory: 16Gi

# SPECIFIC MODEL CONTAINERS

models:

# Container 0 (Inherits global 'vllm' runtime. Port 8080)

- model:

id: "meta-llama/Llama-3.1-8B-Instruct"

maxModelLen: 8192

# Container 1 (Overrides runtime to 'tgi'. Port 8081)

- runtime: "tgi"

model:

id: "google/gemma-3-12b-it"

# Overriding resources for this specific container

resourcesMap:

requests:

cpu: 4

memory: 32Gi

Proxy Configuration for Multi-Model Pods

Because the models now operate on sequential ports (80, 81, 82, etc.) rather than just the default port 80, you must explicitly configure your proxies to route traffic to the correct model port.

API Proxy

Specify the port in the model definition.

models:

- model_name: "llama-3"

model_id: "meta-llama/Llama-3.1-8B-Instruct"

chart: "inference-deployment-name"

port: 80 # Routes to Container 0

- model_name: "gemma-3"

model_id: "google/gemma-3-12b-it"

chart: "inference-deployment-name"

port: 81 # Routes to Container 1

Ingress Proxy

Update the upstream charts list to define both the name and port (if routing directly to the inference engine – TGI use case).

models:

- model: "gemma-3"

charts:

- name: "inference-deployment-name"

port: 81

Architectural Considerations & Warnings

Recommendation: Deploy One Model Per Pod

For production stability, it is highly recommended to deploy one model per pod by using multiple distinct Helm releases (e.g., one release for Llama, a separate release for Gemma) rather than cramming multiple containers into a single Pod.

Deploying multiple models in a single pod introduces several significant challenges:

Independent Scaling is Impossible
The Kubernetes Horizontal Pod Autoscaler (HPA) scales at the Pod level, not the container level. If Model A receives massive traffic and Model B sits idle, the HPA will scale up the entire Pod. This results in deploying unnecessary replicas of Model B, wasting expensive compute and memory resources.

Shared Blast Radius
Containers in a Pod share the same lifecycle. If one model's container crashes due to an Out of Memory (OOM) error or a software panic, it often results in the entire Pod being evicted or restarted. A failure in a low-priority background model will cause downtime for your high-priority production model.

GPU Allocation & Hardware Limitations
This is the most critical hurdle. By default, the Kubernetes NVIDIA Device Plugin provides exclusive access to a physical GPU.
- If Container 0 requests 1 GPU, and Container 1 requests 1 GPU, Kubernetes will attempt to allocate 2 physical GPUs to the Pod.
- If your node only has 1 physical GPU, the Pod will remain in a Pending state indefinitely because Kubernetes cannot natively share a single GPU device across multiple containers.

How to actually share a GPU

To make a multi-model pod share a single physical GPU, your cluster administrators must pre-configure the underlying infrastructure using specific NVIDIA sharing technologies:

NVIDIA MIG (Multi-Instance GPU)
Available on newer architectures (e.g., Ampere A100, Hopper H100). MIG physically partitions the hardware into smaller, isolated virtual GPUs. Kubernetes sees these as separate devices, allowing Container 0 to take "MIG-Profile-A" and Container 1 to take "MIG-Profile-B".

NVIDIA MPS (Multi-Process Service)
A software-level client-server implementation that allows multiple processes to share the GPU's context.

Without MIG or MPS explicitly configured at the cluster level, a multi-model pod will either fail to schedule or require entirely separate physical GPUs for every model within the pod.

Supported/Tested Models and Configurations

This is a list of the latest supported image version and models.

Important: The supported models list below includes only those that have been thoroughly tested by our team. Please note that the actual scope of deployable models is significantly broader.

Supported Models

Hugging Face TGI

Model ID

meta-llama/Llama-3.1-8B-Instruct

vLLM

Model ID

meta-llama/Llama-3.1-8B-Instruct

deepseek-ai/DeepSeek-R1-Distill-Qwen-14B

unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit

unsloth/Qwen3-32B-unsloth-bnb-4bit

google/gemma-3-12b-it

google/gemma-3-27b-it

google/gemma-3-27b-it-qat-q4_0-unquantized

openai/gpt-oss-20b

openai/gpt-oss-120b

deepseek-ai/DeepSeek-OCR

google/gemma-3-12b-it-FP8

google/gemma-3-27b-it-FP8

RedHatAI/gemma-3-12b-it-quantized.w8a8

mistralai/Mistral-Small-3.2-24B-Instruct-2506

RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8

google/gemma-4-31B-it

google/gemma-4-31B

google/gemma-4-26B-A4B-it

google/gemma-4-26B-A4B

google/gemma-4-E4B-it

google/gemma-4-E4B

google/gemma-4-E2B-it

google/gemma-4-E2B

vLLM CPU

Model ID

meta-llama/Llama-3.1-8B-Instruct

SGLang

Model ID

meta-llama/Llama-3.1-8B-Instruct

google/gemma-3-12b-it

google/gemma-3-27b-it

google/gemma-3-27b-it-qat-q4_0-unquantized

openai/gpt-oss-20b

openai/gpt-oss-120b

deepseek-ai/DeepSeek-OCR

Appendix

Helm Charts Configuration

mindbreeze-llm-inference

Global Parameters

Name	Description	Type
instanceCount	Number of pods to deploy. Minimum 2 is recommended for zero-downtime upgrades.	integer
registryPullSecrets	Name of the Kubernetes Secret used to authenticate against a private Docker registry.	string
runtime	Inference engine to run. Possible options: “vllm”, “vllm-cpu”, “sglang”, “tgi” or “audio-default-engine”.	string
runAsThirdPartyDependency	Determines the image naming convention used for the deployment. When set to true (Third-Party Mode): constructs the image as <repository> / <baseName> : <version>. When set to false, appends a predefined image name: <repository> / <baseName> / <predefinedRuntimeImageName> : <version>. Default: false.	boolean
supportLegacyImageDefinition	If true, construct the image as <repository>/<baseName>-<runtime>:<version \| "latest">	boolean
env[].name	Name of the environment variable.	string
env[].value	Value of the environment variable.	string
storageClassName	StorageClass for any PersistantVolumeClaims created by the chart.	string
hub.tokenSecret	Name of the Kubernetes Secret holding your Hugging Face API token.	string

Image Parameters

Name	Description	Type
image.repository	Docker registry (and path) where the image is hosted.	string
image.baseName	Base name of the image (i.e., the application or service).	string
image.version	Tag of the image to deploy.	string
image.digest	Digest of the image to deploy. Takes precedence over version.	string

Model Parameters

Name	Description	Type
model.id	Exact model identifier on Hugging Face (e.g., meta-llama/Llama-3.1-8B-Instruct, facebook/opt-125m).	string
model.permitUnsupportedModels	If true, allows deploying models that haven’t been officially tested.	boolean
model.args	Extra command-line arguments to pass to the model server.	list of strings

Pod Labels

Name	Description	Type
podLabels.gpuLabelName	The label key used to indicate whether a pod is deployed on a GPU-enabled node. Default: “apps.mindbreeze.com/gpu-allocated”.	string

vLLM (CPU)

Name	Description	Type
model.maxNumBatchedTokens	Maximum number of tokens to be processed in a single iteration.	integer
model.maxModelLen	Model context length (prompt and output).	Integer
model.quantization	Quantization format for a pre-quantized model (e.g., int8, fp16).	string
model.loadFormat	The format of the model weights to load.	string
model.reasoningParser	The parser for the reasoning model.	string
model.cpuOffloadGb	Size of the CPU swap space per GPU (in GiB).	float
model.tensorParallelSize	Number of tensor parallel groups.	integer
model.dataParallelSize	Number of data parallel groups.	integer
model.gpuMemoryUtilization	The fraction of GPU memory to be used for the model executor, which can range from 0 to 1.	float [0-1]
model.maxNumSeqs	Maximum number of sequences to be processed in a single iteration.	integer
model.dtype	Data type for model weights and activations.	string
model.logitsProcessors	One or more logits processors’ fully-qualified class names or class definitions.	string
model.enablePrefixCaching	Whether to enable prefix caching. Enabled by default for V1.	boolean
model.noEnablePrefixCaching	Whether to disable prefix caching.	boolean
model.mmProcessorCacheGb	The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs. Set 0 to disable this cache completely (not recommended).	float
model.chatTemplate	Path to the chat template to use.	string
model.disableLogStats	Disable logging statistics.	boolean
model.enableLogRequests	Enable logging requests.	boolean
model.noEnableLogRequests	Disable logging requests.	boolean
model.tokenizerMode	Select tokenizer mode.	string
model.configFormat	The format of the model config to load.	string
model.enableAutoToolChoice	Enable the model to generate its own tool calls when it deems appropriate.	boolean
model.toolCallParser	Select tool call parser.	string
model.disableAccessLogForEndpoints	Disable access logs for specified endpoints. For example, value '/health,/metrics' reduces noise for monitoring systems.	string

SGLang

Name	Description	Type
model.maxTotalTokens	The maximum number of tokens in the memory pool. If not specified, it will be automatically calculated based on the memory usage fraction.	integer
model.quantization	The quantization method.	string
model.loadFormat	The format of the model weights to load.	string
model.reasoningParser	The parser for reasoning models.	string
model.dtype	Data type for model weights and activations.	string
model.maxRunningRequests	The maximum number of running requests.	integer
model.tpSize	The tensor parallelism size.	integer
model.dpSize	The data parallelism size.	integer
model.contextLength	The model's maximum context length.	integer
model.enableMultimodal	Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen.	boolean
model.chunkedPrefillSize	The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill.	integer
model.maxPrefillTokens	The maximum number of tokens in a prefill batch. The real bound will be the maximum of this value and the model's maximum context length.	integer
model.logLevel	The logging level of all loggers.	string
model.logLevelHttp	The logging level of HTTP server.	string
model.logRequests	Log metadata, inputs, outputs of all requests. The verbosity is decided by model.logRequestsLevel.	boolean
model.logRequestsLevel	0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output.	integer

Audio Default Engine

Name

Description

Type

model.audioTranscriptions.models.id

Model ID of the audio transcription model.

Available options: “large-v3”, “large-v3-turbo”, “<other-supported-stt-models>”.

string

Hugging Face TGI

Name	Description	Type
model.maxTotalTokens	The maximum number of tokens in the memory pool. If not specified, it will be automatically calculated based on the memory usage fraction.	integer
model.maxBatchPrefillTokens	Limits the number of tokens for the prefill operation.	integer
model.maxInputTokens	This is the maximum allowed input length (expressed in number of tokens) for users.	integer
model.numShard	The number of shards to use if you don't want to use all GPUs on a given machine.	integer
model.sharded	Whether to shard the model across multiple GPUs.	boolean
model.dtype	The dtype to be forced upon the model. This option cannot be used with 'quantization'.	string
model.cudaMemoryFraction	Limit the CUDA available memory.	float
model.maxConcurrentRequests	The maximum amount of concurrent requests for this particular deployment.	integer
model.quantize	The quantization method.	string

Multiple Models in a Single Pod Deployment Parameters

General

Name	Description	Type
models[].runtime	Inference engine to run. Possible options: “vllm”, “vllm-cpu”, “sglang”, “tgi” or “audio-default-engine”.	string
models[].runAsThirdPartyDependency	Determines the image naming convention used for the deployment. When set to true (Third-Party Mode): constructs the image as <repository> / <baseName> : <version>. When set to false, appends a predefined image name: <repository> / <baseName> / <predefinedRuntimeImageName> : <version>. Default: false.	boolean
models[].supportLegacyImageDefinition	If true, construct the image as <repository>/<baseName>-<runtime>:<version \| "latest">	boolean
models[].env	List of environment variables to set in the container. Each item has name and value fields.	list of objects
models[].env[].name	Name of the environment variable.	string
models[].env[].value	Value of the environment variable.	string

Image Parameters

Name	Description	Type
models[].image	For complete configuration, please refer to Section .	object

Model Parameters

Name	Description	Type
models[].model	For complete configuration, please refer to Section .	object

Audio Default Engine

Name	Description	Type
models[].audioTranscriptions	For complete configuration, please refer to Section .	object

GPU Parameters

Name	Description	Type
models[].gpu	For complete configuration, please refer to Section .	object

Resource Parameters

Name	Description	Type
models[].resourcesMap	For complete configuration, please refer to Section .	object

Startup Probe Parameters

Name	Description	Type
models[].startupProbe	For complete configuration, please refer to Section .	object

Readiness Probe Parameters

Name	Description	Type
models[].readinessProbe	For complete configuration, please refer to Section .	object

Liveness Probe Parameters

Name	Description	Type
models[].livenessProbe	For complete configuration, please refer to Section .	object

GPU Parameters

Name	Description	Type
gpu.enabled	Enable or disable GPU acceleration.	boolean
gpu.count	Number of GPUs to request per pod.	integer
gpu.profile	GPU resource profile (e.g., MIG partition name) to use.	string
gpu.product	GPU SKU to request (e.g., NVIDIA-H100-NVL).	string

Resource Parameters

Name	Description	Type
resourcesMap.storage	Size of the persistent volume claim (e.g., 100Gi).	string
resourcesMap.requests.cpu	CPU request (minimum cores).	integer
resourcesMap.requests.memory	Memory request (e.g., 4Gi).	string
resourcesMap.limits.cpu	CPU limit (maximum cores).	integer
resourcesMap.limits.memory	Memory limit (e.g., 8Gi).	string

Startup Probe Parameters

Name	Description	Type
startupProbe.initialDelaySeconds	Seconds to wait after container start before the first probe.	integer
startupProbe.periodSeconds	Interval (seconds) between startup probes.	integer
startupProbe.failureThreshold	Consecutive failures before marking the pod as not ready.	integer
startupProbe.timeoutSeconds	Number of seconds after which the probe times out. Default: 5.	integer

Readiness Probe Parameters

Name	Description	Type
readinessProbe.initialDelaySeconds	Seconds to wait after container start before the first probe.	integer
readinessProbe.periodSeconds	Interval (seconds) between readiness probes.	integer
readinessProbe.failureThreshold	Consecutive failures before marking the pod as not ready.	integer
readinessProbe.timeoutSeconds	Number of seconds after which the probe times out. Default: 5.	integer

Liveness Probe Parameters

Name	Description	Type
livenessProbe.initialDelaySeconds	Seconds to wait after container start before the first liveness check.	integer
livenessProbe.periodSeconds	Interval (seconds) between liveness checks.	integer
livenessProbe.failureThreshold	Consecutive failures before restarting the container.	integer
livenessProbe.timeoutSeconds	Number of seconds after which the probe times out. Default: 5.	integer

Topology Spread Constraints

Name	Description	Type
topologySpreadConstraints.enabled	Controls if the constraints are applied	boolean
topologySpreadConstraints.constraints	List of topology constraints	list of objects
topologySpreadConstraints.constraints[].maxSkew	Describes the degree to which Pods may be unevenly distributed.	integer
topologySpreadConstraints.constraints[].topologyKey	Key of node labels.	string
topologySpreadConstraints.constraints[].whenUnsatisfiable	Indicates how to deal with a Pod if it doesn't satisfy the spread constraint.	string
topologySpreadConstraints.constraints[].minDomains	Indicates a minimum number of eligible domains.	integer

Network Policies

Name	Description	Type
networkPolicy.enabled	Specifies whether a NetworkPolicy resource should be created for the Inference Engine deployment. Default: false.	boolean
networkPolicy.allowAll	Allows all inbound and outbound traffic for Inference Engine pods in the namespace, completely bypassing any other defined network policy rules. Default: true.	boolean
networkPolicy.dns.enabled	Allows the pod to communicate with the cluster's DNS provider to resolve domain names. Default: false.	boolean
networkPolicy.dns.allowExternal	Specifies whether external (outside the cluster) DNS resolution is permitted. Default: false.	boolean
networkPolicy.ingressProxy.enabled	Allows incoming (ingress) traffic originating from the ingress proxy controller (e.g., NGINX). Default: false.	boolean
networkPolicy.ingressProxy.ports	A list of destination ports to allow incoming traffic from the ingress proxy. Default: [8080].	list of integers
networkPolicy.apiProxy.enabled	Allows incoming (ingress) traffic originating from the API Proxy (e.g., LiteLLM). Default: false.	boolean
networkPolicy.apiProxy.ports	A list of destination ports to allow incoming traffic from the API Proxy. Default: [8080].	list of integers
networkPolicy.internetAccess.enabled	Allows the pod to make outbound (egress) connections to the internet. Default: false.	boolean
networkPolicy.internetAccess.onlyHTTPS	Restricts outbound internet connections to only allow HTTPS traffic (port 443). Default: false.	boolean
networkPolicy.internetAccess.ports	A list of specific custom ports permitted for outbound internet access.	list of integers
networkPolicy.httpProxy.enabled	Allows outgoing (egress) traffic to the HTTP proxy. Default: false.	boolean
networkPolicy.httpProxy.url	The URL endpoint used to route traffic to HTTP proxy.	string
networkPolicy.httpProxy.port	The network port required to communicate with HTTP proxy. Default: 80.	integer
networkPolicy.httpProxy.address	Specific IP addresses or CIDR blocks associated with the HTTP proxy.	list
networkPolicy.additionalIngressRules	Allows defining custom Kubernetes NetworkPolicy ingress rules to append to the generated policy.	list of objects
networkPolicy.additionalEgressRules	Allows defining custom Kubernetes NetworkPolicy egress rules to append to the generated policy.	list of objects

mindbreeze-api-llm-proxy

Global Parameters

Name	Description	Type
instanceCount	Number of pods to deploy. Minimum 2 is recommended for zero-downtime upgrades.	integer
registryPullSecrets	Name of the Kubernetes Secret used to authenticate against a private Docker registry.	string
env	List of environment variables to set in the container. Each item has name and value fields.	string
env[].name	Name of the environment variable.	string
env[].value	Value of the environment variable.	string
skip_ssl_verification	If true, disables SSL certificate verification calls.	boolean
implementation	LiteLLM settings schema version used by the Helm config template. Default: litellm-1.	string

Image Parameters

Name	Description	Type
image.repository	Docker registry (and path) where the image is hosted.	string
image.baseName	Base name of the image (i.e., the application or service).	string
image.version	Tag or digest of the image to deploy.	string
image.digest	Digest of the image to deploy. Takes precedence over version.	string

Model Routing Parameters

Name	Description	Type
models	List of routing rules, each defining which LLM deployment handles a given request.	list of objects
models[].model	Logical name of the model exposed via the OpenAI-compatible API.	string
models[].model_id	Actual ID of the backend model being served (must match your deployment).	string
models[].chart	Alias of the Helm release (or sub-chart) where this model is deployed.	string
models[].port	The port number on which the inference engine serves the model. Overrides the default if specified. Default: 80.	integer
models[].requests_per_minute	Maximum number of requests this model can handle per minute.	integer
models[].tokens_per_minute	Maximum total input + output tokens processed per minute.	integer
models[].required_roles	List of user roles authorized to access this model.	list of strings

Authentication and impersonation parameters

Name	Description	Type
auth.enabled	Enable or disable authentication. When disabled, keycloak configuration is not used.	boolean
auth.impersonation.enabled	Enables header-based impersonation. Requires auth.enabled: true.	boolean
auth.impersonation.headers.organization.name	HTTP header from which the organization ID is read.	string
auth.impersonation.headers.organization.allowedPattern	Regex the organization ID header value must fully match. null disables validation.	string or null
auth.impersonation.headers.team.name	HTTP header from which the team ID is read.	string
auth.impersonation.headers.team.allowedPattern	Regex the team ID header value must fully match. null disables validation.	string or null
auth.impersonation.headers.user.name	HTTP header from which the user ID is read.	string
auth.impersonation.headers.user.allowedPattern	Regex the user ID header value must fully match. null disables validation.	string or null

Keycloak Parameters

Name	Description	Type
keycloak.useConfigMap	Use ConfigMap instead of secret to configure keycloak authentication.	boolean
keycloak.configMap	ConfigMap name, which contains a keycloak realm and a key. Only valid when useConfigMap is true.	string
keycloak.secret	Secret name, which contains a keycloak realm and a key.	string
keycloak.realmKey	Secret key, which holds a keycloak realm.	string
keycloak.urlKey	Secret key, which holds a keycloak url.	string

Topology Spread Constraints

Name	Description	Type
topologySpreadConstraints.enabled	Controls if the constraints are applied	boolean
topologySpreadConstraints.constraints	List of topology constraints	list of objects
topologySpreadConstraints.constraints[].maxSkew	Describes the degree to which Pods may be unevenly distributed.	integer
topologySpreadConstraints.constraints[].topologyKey	Key of node labels.	string
topologySpreadConstraints.constraints[].whenUnsatisfiable	Indicates how to deal with a Pod if it doesn't satisfy the spread constraint.	string
topologySpreadConstraints.constraints[].minDomains	Indicates a minimum number of eligible domains.	integer

Network Policies

Name	Description	Type
networkPolicy.enabled	Specified whether a NetworkPolicy resources should be created for the API LLM Proxy. Default: false.	boolean
networkPolicy.allowAll	Allows all inbound and outbound traffic for API LLM Proxy pods in the namespace, completely bypassing any other defined network policy rules. Default: true.	boolean
networkPolicy.dns.enabled	Allows the pod to communicate with the cluster’s DNS provider to resolve domain names. Default: false.	boolean
networkPolicy.dns.allowExternal	Specifies whether external (outside the cluster) DNS resolution is permitted. Default: false.	boolean
networkPolicy.ingressProxy.enabled	Allows incoming (ingress) traffic originating from the ingress proxy (e.g., NGINX). Default: false.	boolean
networkPolicy.keycloak.enabled	Allows outgoing (egress) traffic to the Keycloak authentication service. Default: false.	boolean
networkPolicy.keycloak.url	The URL endpoint used to route traffic to Keycloak.	string
networkPolicy.keycloak.port	The network port required to communicate with Keycloak. Default: 80.	integer
networkPolicy.keycloak.address	Specific IP addresses or CIDR blocks associated with the Keycloak instance.	list
networkPolicy.inferenceEngine.enabled	Allows outgoing (egress) traffic to the Inference Engine. Default: false.	boolean
networkPolicy.inferenceEngine.ports	A list of destination ports to allow traffic to the Inference Engine. Default: [8080].	list of integers
networkPolicy.additionalIngressRules	Allows defining custom Kubernetes NetworkPolicy ingress rules to append to the generated policy.	list of objects
networkPolicy.additionalEgressRules	Allows defining custom Kubernetes NetworkPolicy egress rules to append to the generated policy.	list of objects

mindbreeze-llm-proxy

Global Parameters

Name	Description	Type
instanceCount	Number of pods to deploy. Minimum 2 is recommended for zero-downtime upgrades.	integer
registryPullSecrets	Name of the Kubernetes Secret for authenticating with a private Docker registry.	string
clusterExternalDomain	External domain of the OpenShift cluster where this application is hosted.	string
Env	List of environment variables to set in the container. Each item must include name and value.	string
env[].name	Name of the environment variable.	string
env[].value	Value of the environment variable.	string
ingressTLSSecret	Name of the Kubernetes Secret containing the TLS certificate for securing Ingress traffic.	string
ingressTimeout	Timeout in seconds for Ingress requests/responses.	integer
Endpoint	URL path or hostname where the API will be exposed.	string
supportLegacyUnauthenticatedTGI	If true, enables support for legacy unauthenticated TGI endpoints.	boolean

Image Parameters

Name	Description	Type
image.repository	Docker registry (and path) where the image is stored.	string
image.baseName	Base name of the image (i.e., the application or service).	string
image.version	Tag or digest of the image to deploy.	string
image.digest	Digest of the image to deploy. Takes precedence over version.	string

Model Routing Parameters

Name	Description	Type
models	List of routing rules mapping each endpoint to one or more LLM service deployments. Requests are round-robin among services.	list of objects
models[].model	URL endpoint exposed for this model (used in the OpenAI-compatible API).	string
models[].charts	A list of Helm release aliases (sub-charts) associated with this endpoint. By default, it is assumed the inference engine serves traffic on port 80.	list of strings, or list of objects (see below)
models[].charts[].name	The specific name or alias of the model deployment release.	string
models[].charts[].port	The port number on which the inference engine serves the model. Overrides the default if specified. Default: 80.	integer

API Proxy Parameters

Name	Description	Type
apiProxy.charts	List of Helm release aliases for the API LLM proxy services.	list of strings

Proxy limitation parameters

Name	Description	Type
limits.workerConnections	Maximum simultaneous connections per worker process.	integer
limits.connectionsPerUser	Maximum simultaneous connections allowed per user.	integer
limits.connectionsGlobal	Maximum total simultaneous connections across all users.	integer
limits.keepAliveTimeout	How long an idle keep-alive connection with a client is kept open.	integer or string
limits.mapHashBucketSize	Sets the size of the hash bucket for the “map” directive.	integer or string
limits.clientBodyBufferSize	Maximum size of the client request body that will be buffered in memory.	integer or string
limits.clientMaxRequestBodySize	Maximum allowed request body size.	integer or string
limits.proxyReadTimeout	Defines a timeout for reading a response from the proxied server. Default: 60 seconds.	integer
limits.proxySendTimeout	Sets a timeout for transmitting a request to the proxied server. Default: 60 seconds.	integer
limits.proxyConnectTimeout	Defines a timeout for establishing a connection with a proxied server.	integer

Ingress Parameters

Name	Description	Type
ingress.enabled	If true, create an Ingress resource for the service.	boolean

NGINX Parameters

Name	Description	Type
nginx.useHTTPS	If true, serve traffic over HTTPS in the NGINX proxy.	boolean
nginx.TLSSecret	Name of the Kubernetes Secret containing TLS certificates for NGINX.	string

Service Parameters

Name	Description	Type
service.type	Kubernetes Service type: ClusterIP, NodePort, LoadBalancer, or ExternalName.	string
service.ports	List of ports exposed by the Service. Each entry requires name, port, targetPort, and protocol.	list of objects
service.ports[].name	Identifier for the port.	string
service.ports[].port	Port number exposed by the Service.	integer
service.ports[].targetPort	Port number on the pod to which traffic is directed.	integer
service.ports[].protocol	Network protocol (e.g., TCP, UDP).	string
service.externalName	DNS name to route to when service.type is ExternalName.	string
service.loadBalancerIP	Static IP to assign for a LoadBalancer Service.	string
service.clusterIP	Static internal IP for a ClusterIP Service; must lie within the cluster’s service CIDR.	string
service.externalIPs	List of external IP addresses that forward traffic to this Service.	list of strings
service.labels	Additional labels to attach to the Service metadata.	list of key-value pairs
service.annotations	Annotations for external integrations (e.g., monitoring, service mesh).	list of key-value pairs
service.selector	Label selector to identify pods targeted by the Service.	list of key-value pairs
service.status	Read-only status fields managed by Kubernetes; not user-defined.	object

Logging Parameters

Name	Description	Type
logging.label	Key used to tag and identify log entries.	string
logging.schema	Name of the log schema applied by this chart’s pods.	string

Topology Spread Constraints

Name	Description	Type
topologySpreadConstraints.enabled	Controls if the constraints are applied	boolean
topologySpreadConstraints.constraints	List of topology constraints	list of objects
topologySpreadConstraints.constraints[].maxSkew	Describes the degree to which Pods may be unevenly distributed.	integer
topologySpreadConstraints.constraints[].topologyKey	Key of node labels.	string
topologySpreadConstraints.constraints[].whenUnsatisfiable	Indicates how to deal with a Pod if it doesn't satisfy the spread constraint.	string
topologySpreadConstraints.constraints[].minDomains	Indicates a minimum number of eligible domains.	integer

Network Policies

Name	Description	Type
networkPolicy.enabled	Specifies whether a NetworkPolicy resource should be created for the Ingress Proxy. Default: false.	boolean
networkPolicy.allowAll	Allows all inbound and outbound traffic for Ingress Proxy pods in the namespace, completely bypassing any other defined network policy rules. Default: true.	boolean
networkPolicy.dns.enabled	Allows the pod to communicate with the cluster's DNS provider to resolve domain names. Default: false.	boolean
networkPolicy.dns.allowExternal	Specifies whether external (outside the cluster) DNS resolution is permitted. Default: false.	boolean
networkPolicy.ingressController.enabled	Allows incoming (ingress) traffic originating from the OpenShift router. Default: false.	boolean
networkPolicy.ingressController.namespaceSelectors	Defines the specific namespace selectors from which OpenShift router traffic is allowed.	list of objects
networkPolicy.hostNetwork.enabled	Allows incoming (ingress) traffic originating from the OpenShift host network. Default: false.	boolean
networkPolicy.hostNetwork.namespaceSelectors	Defines the specific namespace selectors from which OpenShift host network traffic is allowed.	list of objects
networkPolicy.inferenceEngine.enabled	Allows outgoing (egress) traffic to the Inference Engine. Default: false.	boolean
networkPolicy.inferenceEngine.ports	A list of destination ports to allow traffic to the Inference Engine. Default: [8080].	list of integers
networkPolicy.apiProxy.enabled	Allows outgoing (egress) traffic to the API Proxy (e.g., LiteLLM). Default: false.	boolean
networkPolicy.externalAccess.enabled	Allows external network access to the application. Default: false.	boolean
networkPolicy.externalAccess.allowedCIDRs	A list of approved IP ranges (CIDR blocks) that are permitted external access. Default: [“0.0.0.0/0”].	list of strings
networkPolicy.additionalIngressRules	Allows defining custom Kubernetes NetworkPolicy ingress rules to append to the generated policy.	list of objects
networkPolicy.additionalEgressRules	Allows defining custom Kubernetes NetworkPolicy egress rules to append to the generated policy.	list of objects

{{{i18n.refineSearch}}}

Whitepaper Mindbreeze InSpire LLM: A Kubernetes Integration Guide

Introduction

Chart Template

Mindbreeze LLM Inference

Deployment

Configuration

Example Configuration

Essential Configuration Elements

Mindbreeze LLM Proxy

Deployment

Configuration

Example Configuration

Essential Configuration Elements

URL Structure

Limits – Default Values

Creating a Secret

Authentication

Keycloak

Keycloak Configuration Example

Creating a Realm

Creating a client

Assigning roles to a client

Authentication Proxy Deployment

Example: Deploying Authentication proxy

Add dependencies

Proxy configuration

Authentication Proxy Configuration

Configuration Parameters

Creating a Keycloak Configuration

ConfigMap

Secret

RAG Service Integration

Creating credentials

Creating an LLM in RAG service

OpenAI compatible Model Configuration

Creating an OpenAI compatible Model

Creating an InSpire LLM Model

Using Huggingface TGI API (Legacy)

Configuring TGI Model in RAG

Disabling Authentication

Use Cases

Role-based Access Control

Model Discovery & Registry

Rate-limiting & Cost Management

Rate-limiting

Cost Management

Guardrails: Default & Custom Enforcement

Header-based Impersonation & Webhook Integration

Configuring impersonation headers

Parameters

Examples

Registering a webhook

Using impersonation data in a webhook

Network Policies

Architecture & Traffic Flow

Global Configurations

Enabling Network Policies

The “Allow All” Override

DNS Resolution

Common Configuration Use Cases

Scenario A: Enabling the Standard Multi-Tier Flow

Scenario B: Enabling Internet Access for Model Downloads

Scenario C: Connecting API Proxy to Keycloak

Scenario D: Allowing External Traffic into the Ingress Proxy

Advanced Rules

Scenarios

Deploying Llama-3.1-8B-instruct with Hugging Face TGI using ArgoCD and exposing it externally

Clone the repository

Add dependencies

Apply the deployment configuration

Push the changes

Notes

Access URL

Setting a Specific GPU Device

Deploying a Model on CPU

Notes on CPU Deployments

Deploying and Using Speech-to-Text (STT) Models

Deploying the STT Model

Making Requests (Speaches via Proxy)

Whitepaper
Mindbreeze InSpire LLM: A Kubernetes Integration Guide