Whitepaper
Mindbreeze InSpire LLM: On-Premise Deployment Guide

Introduction

Mindbreeze InSpire LLM lets you run large language models directly on a GPU-enabled InSpire appliance, entirely within your own infrastructure. Rather than routing requests to an external, hosted provider, models are deployed, served, and consumed on-premise – keeping prompts, source documents, and generated output inside your own security and network boundary, and giving you full control over which models run and who may use them.

On-premise deployments are managed through the InSpire LLM CLI (inspire-llm), which drives the full lifecycle of a model on the appliance. The CLI is built around three concepts:

• Template – a base configuration for deploying a particular model.
• Provider – a parameterized template that fixes the GPU(s) device(s), the maximum token limit, whether the model is published (production) or unpublished (testing/development), and custom deployment arguments (e.g., GPU utilization).
• Container – the Docker container that actually runs the model built from a chosen provider.

Once started, a model is exposed over authenticated HTTPS endpoints on the appliance and can be consumed directly or through the InSpire RAG service. Access is secured with Keycloak using OAuth 2.0 / JWT tokens and role-based authorization, so only holders of the appropriate service-account roles – InSpire LLM Service User for published models, InSpire LLM Unpublished Services User for unpublished ones – can reach a given model.

This guide walks through everything required to stand up such a deployment: preparing the GPU host and NVIDIA driver container, configuring the Keycloak service account, installing the InSpire LLM package, and creating, starting, accessing, and cleaning up model deployments — with CLI reference, health checks, and troubleshooting in the appendix.

Prerequisites

The InSpire LLM container can be deployed on a GPU-enabled Hardware, where the NVIDIA driver container is up to date.

Installation process of the NVIDIA driver container

The correct NVIDIA driver container image matching the kernel version is required:

1. Install the image before updating to an InSpire version with a new kernel.

1. Install the update and the driver will be loaded automatically at boot.

If the driver for the current kernel is not installed, the driver can also be loaded manually (after loading the image) to be available without reboot.

Setup of Service Account

The HTTPD configuration allows only users with the role “InSpire LLM Services User” and “InSpire LLM Unpublished Services User” using OAuth2.0/JWT-tokens to access the LLMs. The difference between those roles is:

• “InSpire LLM Services User” can only use published model providers.
• “InSpire LLM Unpublished Services User” may also use unpublished models.

Additionally, the user name has to be part of the URL. For example:

Currently, the Chat Service uses the service account of the client, therefore a client has to be created for each customer instead of a user.

Prepare Realm Role

In the Mindbreeze InSpire MMC Console navigate to “Setup” and then “Credentials”.

Please create the realm role in the Keycloak realm where the service accounts will be created (default realm is “master”):

• “InSpire LLM Services User”

Add OIDC Client for the Service Account

Please add the client in the “Clients” section:

Setup of Client Parameters

The OIDC Client should be enabled for “Client Authentication” and support the “Service Accounts roles” Authentication Flow.

Both settings can be configured during client creation as shown in the screenshot or just be set after the client is configured.

Assigning Service Account Role

On the “Service Account Roles” tab of the OIDC client settings, add the previously created “InSpire RAG Impersonating User” role to the service account of the client.

Exporting Realm Access Roles in the OAuth Access Tokens

For checking the required role for impersonation, the Mindbreeze InSpire RAG Service is expecting the realm roles as “roles” attribute of the OAuth access token.

This requires the following configuration steps on the previously created OIDC client:

1. Navigate to the “Client scopes” tab and open the settings of the scope marked as “Dedicated scope and mappers for this client”.

1. Add the “realm roles” mapper from the “Predefined Mappers” by clicking on “Add Mapper” and selecting the “From Predefined Mappers” mapper type.

1. Edit the added mapper, and change the Token Claim Name attribute to “roles” so that the service user realm roles are written as “roles” claim in the access token.

1. Save your changes

Installation Process

Prerequisites

Before installing the InSpire LLM Installer (version 25.6), it is recommended to remove any previous InSpire LLM installations to ensure a clean environment. Follow these steps to remove existing components:

1. Remove system service files:

1. Stop and remove all running InSpire LLM Docker containers:

1. Delete the InSpire LLM installation directory (default path: /var/data/inspire-llm):

Installation

To install InSpire LLM, follow the steps below:

1. Copy the installation archive to the target host, typically under /var/data.
2. Set the installer as executable:

1. Execute the installer:

Note:

During installation, all packaged Docker images and LLM models will be extracted to /var/data/inspire-llm by default. If you want to change the destination directory, execute the installer the following way:

(Optional) Make the InSpire CLI Globally Available

By default, the InSpire LLM CLI script is located at:

/var/data/inspire-llm/bin/inspire-llm.sh

To avoid specifying the full path every time you use the CLI, you can create a symbolic link in /usr/local/bin. This will allow you to run the CLI from any directory.

To the create the symbolic link:

After creating the symbolic link, you can use the InSpire LLM CLI simply by running:

from any location in your terminal.

Deploying a Large Language Model

Creating a Provider

InSpire LLM uses the concepts of templates and providers:

• A template is a base configuration used to deploy a large language model.
• A provider is a parameterized template that specified:
  • The GPU device(s) for deployment
  • The maximum allowed tokens for the model
  • Whether the model is published or unpublished

You can create a provider either through the CLI (with options) or interactively:

1. List available templates:

Copy the desired template name from the output

1. Create a provider interactively:

Or,

Create a provider non-interactively:

1. Verify provider creation:

Your new provider should appear in the list.

Deploying a Single Model on Multiple GPUs

To deploy a single model across multiple GPUs, specify the -g option with a comma-separated list of GPU device IDs. For example:

Published vs. Unpublished Providers

• A published provider exposes the model at the endpoint:

• An unpublished provider exposes the model at:

Use published for production deployments and unpublished for testing and development environments.

Creating a Container for the LLM

Once you have created a provider, you can create a container to run the LLM.

1. List available providers:

Copy the provider name you want to use.

1. Create a container:

This will create a Docker container for the specified provider.

Starting a Large Language Model

After creating a container, you can start it:

1. List available containers:

Copy the container name.

1. Start the container:

This will start the LLM container and refresh proxy configurations.

1. Verify deployment:

Check the container logs to ensure the server started successfully:

Note: Starting a large language model may take several minutes.

Cleaning Up Deployments

InSpire LLM provides commands to stop and delete deployments when needed.

1. List running containers:

Copy the name of the container you wish to stop or delete.

1. Stop a container:

1. Delete a container:

If the container is running, it will be stopped, the proxy configuration will be refreshed, and then the container will be deleted.

1. Delete a provider (and its containers):

This will stop and delete any associated containers, refresh the proxy configuration, and remove the provider.

Note:

• Always ensure you use the correct names for templates, providers, and containers as listed by the inspire-llm list, inspire-llm available-containers and inspire-llm running-containers commands.
• Published models should be used for production endpoints; unpublished for development and testing.

Accessing a Deployed Model

After deploying a model, it is accessible via HTTP endpoints:

• If the InSpire container is running on your local machine:

Or, if unpublished:

(Replace the <appliance-fqdn> with the appliance hostname you have deployed a model on)

• If you are accessing through the proxy on the machine:

Or, if unpublished:

Note:

• <customer-user-name> may be any value.

Appendix

InSpire LLM CLI Commands

LLM Deployment Health Checks

You can verify the health and status of your LLM deployments using one of the following endpoints. The correct URL depends on how your deployment is exposed.

InSpire Hostname

If your LLM deployment is exposed through a valid InSpire container, use the following URL:

Direct host or IP address

If you did not use InSpire to expose your deployment and are connecting directly to the host or IP address, use this URL:

Reverse Proxy

If you are checking the health from inside a machine utilizing the reverse proxy, use this URL:

Note: Authorization is not required to access the health check endpoints in any of the scenarios listed above.

Potential Errors

CUDA is out of memory

If the model is deployed on a machine which has not enough GPU memory space, starting a container would cause it to constantly restart. When typing:

we could see the error “CUDA is out of memory”. In such case, the installation should be done on a machine which has substantial GPU memory space.

Proxy Service Fails to Run

If we inspect the status of the llm-proxy service, which is running on the host machine:

and the result is something like this:

llm-proxy.service: Failed to load environment files: Permission denied
llm-proxy.service: Failed to run 'start-pre' task: Permission denied
llm-proxy.service: Failed with result 'resources'.

We could solve this by setting the SELinux to permissive mode:

To verify that it is permissive type:

After this we can restart the llm-proxy service and it should be running: