Skip to content

REDHAT_AI.md

Latest commit

 

History

History
215 lines (160 loc) · 6.41 KB

REDHAT_AI.md

File metadata and controls

215 lines (160 loc) · 6.41 KB
Raw

Red Hat AI Provider

Abbenay includes a dedicated redhat engine for connecting to Red Hat AI inference endpoints. It supports two deployment profiles that both expose an OpenAI-compatible REST API:

Profile Product Typical endpoint
Inference Server Red Hat AI Inference (vLLM) http://127.0.0.1:8000/v1
MaaS Red Hat OpenShift AI Models-as-a-Service https://maas.apps.cluster.example.com/v1

Both speak the same protocol, so Abbenay routes them through @ai-sdk/openai-compatible — no additional SDK packages are required.

Local Inference Server deployments support air-gapped / offline inference (prompts stay on your network). That does not mean network isolation secures the daemon — see SECURITY.md.

Profile A — Inference Server

Red Hat AI Inference is a standalone vLLM-based inference stack running locally, on a RHEL AI host, or in an OpenShift container.

Quick start

  1. Start the server — it must serve at least one model at the default local endpoint http://127.0.0.1:8000/v1:

    curl http://127.0.0.1:8000/v1/models

  2. Configure Abbenay (~/.config/abbenay/config.yaml on Linux, ~/Library/Application Support/abbenay/config.yaml on macOS):

    providers:
  redhat-inference:
    engine: redhat
    models:
      RedHatAI/Llama-3.2-1B-Instruct-FP8: {}

    Replace the model ID with whatever your server's /v1/models reports.

  3. Chat:

    aby start
aby chat -m redhat-inference/RedHatAI/Llama-3.2-1B-Instruct-FP8

Authentication

Inference Server auth is optional — it depends on whether the server was started with the --api-key flag.

No key (default):

providers:
  redhat-inference:
    engine: redhat
    models:
      RedHatAI/Llama-3.2-1B-Instruct-FP8: {}

With key:

providers:
  redhat-inference:
    engine: redhat
    api_key_env_var_name: "REDHAT_AI_API_KEY"
    models:
      RedHatAI/Llama-3.2-1B-Instruct-FP8: {}
export REDHAT_AI_API_KEY="your-server-api-key"

Custom base URL

Override the default for non-standard ports, remote hosts, or OpenShift routes:

providers:
  redhat-inference:
    engine: redhat
    base_url: "http://192.168.1.100:8080/v1"
    models:
      RedHatAI/Llama-3.2-1B-Instruct-FP8: {}

For servers with self-signed or internal CA certificates:

export NODE_EXTRA_CA_CERTS="/path/to/ca-bundle.crt"

Profile B — OpenShift AI MaaS

OpenShift AI 3.4 delivers Models-as-a-Service (MaaS): a centrally governed, self-service AI gateway with token quotas, rate limiting, and usage tracking. MaaS exposes an OpenAI-compatible /v1/chat/completions endpoint that can route to locally hosted models (vLLM) or external providers.

Configuration

MaaS typically requires an API key (self-service MaaS keys issued by the platform):

providers:
  redhat-maas:
    engine: redhat
    base_url: "https://maas.apps.cluster.example.com/v1"
    api_key_env_var_name: "REDHAT_AI_API_KEY"
    models:
      llama-3.1-8b-instruct: {}
export REDHAT_AI_API_KEY="your-maas-api-key"

Note: The base_url must point to the OpenAI-compatible inference endpoint (/v1), not the native MaaS management API (/maas-api/v1). The /maas-api/v1 prefix is used for token generation and subscription-level model listing — Abbenay does not use it.

Key differences from Inference Server

Aspect Inference Server MaaS
Auth Optional (--api-key) Typically required (self-service keys)
Base URL http://127.0.0.1:8000/v1 Cluster route, e.g. https://maas.apps.…/v1
Governance None (bare metal) Token quotas, rate limiting, showback
Model routing Single model Gateway can route to multiple backends

Model Discovery

aby list-models --discover redhat

# Or directly:
curl http://127.0.0.1:8000/v1/models

The web dashboard Add Provider wizard also supports model discovery when you select the redhat engine.

Tool Calling

Tool calling support depends on the model being served. Models like Llama 3.2 Instruct support tool calling; others may not. If the served model does not support tools, chat will still work but tool calls will not be executed.

Container Usage

When running Abbenay inside a container and the Inference Server is on the host:

providers:
  redhat-inference:
    engine: redhat
    base_url: "http://host.containers.internal:8000/v1"
    models:
      RedHatAI/Llama-3.2-1B-Instruct-FP8: {}

Validation Checklist

# Scenario How to test
1 Auth (no key) Start server without --api-key; omit key fields
2 Auth (with key) Start server with --api-key; set REDHAT_AI_API_KEY
3 Missing key Omit key when server requires it; expect 401
4 MaaS endpoint Set base_url to MaaS route; provide API key
5 Model discovery (CLI) aby list-models --discover redhat
6 Model discovery (web) Add Provider wizard → select redhat
7 Chat completions aby chat -m redhat-inference/<model>
8 Streaming curl -H "Authorization: Bearer $ABBENAY_API_TOKEN" http://127.0.0.1:8787/v1/chat/completions with "stream": true
9 Tool calling aby chat with tools enabled (if model supports it)
10 VS Code LM API Configure Provider → select model → chat
11 Air-gap / privacy Use local Inference Server only; isolation alone does not secure Abbenay — see SECURITY.md

References