Heimdall LLM Context

Purpose
Heimdall is an authenticated local compatibility router. It exposes a small OpenAI-style subset for chat and embeddings, but it is not a model host and not a full vendor API.

Terminology
- Heimdall: authenticated local HTTP compatibility router.
- Shim: small compatibility layer that makes one interface look like another.
- Router: component that chooses the backend for a request, here based on URL path.
- Node proxy: Heimdall front door; handles auth, routing, metrics, chat adaptation, and embeddings proxying.
- Chat adapter: route that translates POST /v1/chat/completions into a local claude -p invocation.
- Embeddings backend: private FastAPI/FastEmbed service that generates embedding vectors.
- Embedding: numeric vector used for search, matching, and similarity.

Scope and contract
- Clients call Heimdall, not private local backends.
- Every route requires Authorization: Bearer <token>.
- Token values and credential locations are intentionally omitted.
- Heimdall authenticates the request, chooses a backend by route, adapts the request if needed, runs or proxies the local backend, and normalizes the response.
- The stable contract is the subset documented here.

Deployment shape
- Source is maintained in a private Heimdall project checkout on the server; exact filesystem paths are intentionally omitted.
- Node front door: authenticated HTTP proxy that owns auth, routing, metrics, chat adaptation, and embeddings proxying.
- Embeddings backend: private FastAPI/FastEmbed process reached only through the Node front door.
- Local bindings, ports, env files, service users, and account identifiers are private operational details.
- This summary intentionally omits secrets, token values, private credential paths, service users, and private identifiers.

Endpoint: POST /v1/chat/completions
Purpose: API-shaped chat responses.
Backend: current local chat adapter, backed by Claude Code CLI through the Node front door.
Behavior:
- Requires Authorization: Bearer <token>.
- Accepts an OpenAI-style chat-completions request shape.
- Reads messages, stream, response_format, and selected heimdall controls.
- Converts the first system message into a Claude Code system prompt.
- Joins user messages into stdin.
- Runs claude -p locally.
- Maps the result back into an API-style chat response.

Minimal chat request:
POST /v1/chat/completions
Authorization: Bearer <token>
Content-Type: application/json

{
  "model": "sonnet",
  "stream": false,
  "messages": [
    { "role": "system", "content": "Return concise JSON." },
    { "role": "user", "content": "Extract the key facts." }
  ],
  "response_format": { "type": "json_object" },
  "heimdall": { "max_turns": 3, "timeout_ms": 180000 }
}

Typical non-streaming chat response shape:
{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "choices": [
    { "index": 0, "message": { "role": "assistant", "content": "..." }, "finish_reason": "stop" }
  ],
  "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0 }
}

Endpoint: POST /v1/embeddings
Purpose: local embedding vectors. Embeddings are numeric vectors used for search, matching, and similarity.
Backend: private FastAPI/FastEmbed service.
Behavior:
- Requires Authorization: Bearer <token> at the Node front door.
- Does not invoke Claude Code.
- Forwards the request body to the local embeddings backend.
- Uses FastEmbed with BAAI/bge-small-en-v1.5.
- Returns 384-dimensional embeddings.

Minimal embeddings request:
POST /v1/embeddings
Authorization: Bearer <token>
Content-Type: application/json

{
  "model": "BAAI/bge-small-en-v1.5",
  "input": ["text to embed"]
}

Typical embeddings response shape:
{
  "object": "list",
  "model": "BAAI/bge-small-en-v1.5",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.01, -0.02, "..."] }
  ],
  "usage": { "prompt_tokens": 0, "total_tokens": 0 }
}

Request flow
1. Client sends an HTTP request with a bearer token.
2. Node front door validates auth and records request metadata.
3. URL path selects the backend.
4. Heimdall adapts the request for that backend.
5. The local backend performs the work.
6. Heimdall returns an API-shaped response or an API-shaped error.

Chat details
- Chat is the current local chat adapter.
- The route is POST /v1/chat/completions.
- Claude Code is invoked through claude -p.
- Chat requests are queued with limited concurrency so the host is not saturated.
- stream=true returns server-sent-event heartbeats plus a final response, not true token-by-token streaming.
- response_format.type=json_object asks Heimdall to enforce and extract JSON object output.
- response_format.type=json_schema asks Heimdall to pass a schema to Claude Code and return structured output.
- heimdall.max_turns and heimdall.timeout_ms are supported but clamped by service policy.
- model, max_tokens, and temperature are accepted for compatibility but are not fully wired through to Claude Code behavior yet.
- Multi-turn conversations are flattened; system and user content are what matter for the Claude Code invocation.

Chat runtime defaults from source
- MAX_CONCURRENT default: 2 active chat requests.
- QUEUE_MAX default: 16 waiting chat requests.
- QUEUE_TIMEOUT_MS default: 90000 ms.
- REQUEST_TIMEOUT_MS default: 300000 ms.
- heimdall.timeout_ms is clamped between 1000 ms and MAX_REQUEST_TIMEOUT_MS.
- heimdall.max_turns is clamped between 1 and 5 by default.
- DEFAULT_TEXT_MAX_TURNS default: 1.
- DEFAULT_SCHEMA_MAX_TURNS default: 3.

Embeddings details
- Embeddings are a separate backend behind the same authenticated shim.
- The route is POST /v1/embeddings.
- The backend is local FastAPI using FastEmbed.
- The model is BAAI/bge-small-en-v1.5.
- The output dimension is 384.
- The embeddings backend is private to the host and is reached through the authenticated Node front door.
- MAX_INPUTS default: 32.
- MAX_TEXT_LENGTH default: 16000 characters per input.

Security model
- Every public Heimdall route requires Authorization: Bearer <token>.
- The token is host-local runtime state and is not documented here.
- Heimdall services listen on loopback addresses.
- Chat and embeddings run as separate systemd services.
- The service units use restricted write paths, no ambient capabilities, restart-on-failure behavior, and journald logging.

Client contract
- Always send Authorization: Bearer <token>.
- Use POST /v1/chat/completions for the current local chat adapter.
- Use POST /v1/embeddings for local embedding vectors.
- Treat operational status and metrics routes as private operator surfaces, not public client contract.
- Treat Heimdall as a narrow compatibility contract, not as a complete provider API.
- Do not rely on unsupported provider fields unless the Heimdall README documents them.
- New routes, backends, and request controls should be documented before clients depend on them.

Main caveats
- Chat latency can be high because each chat call starts a local CLI process and may run for minutes.
- Chat concurrency is intentionally low and queued.
- Overload can return 503 when the queue is full or times out.
- Streaming is heartbeat plus final response.
- Heimdall is useful for controlled internal workloads, not as a general production inference platform.

Error behavior from source
- 401 auth_error: missing, empty, or mismatched bearer token.
- 404 Not found: unsupported route.
- 499 proxy_error: client disconnected during chat work.
- 500 proxy_error: chat proxy or local CLI failure.
- 503 concurrency_limit: chat queue wait timed out.
- 503 queue_full: chat queue is full.
- 503 embeddings_proxy_error: embeddings backend proxy failure.
- 504 proxy_error: chat request timed out.
- Embeddings backend can return 400 for malformed input, 413 for too many inputs or too-long input, and 500 for backend failures.