Triton Inference Server Provider

Serves models through NVIDIA Triton's two cooperating HTTP surfaces — an OpenAI-compatible chat frontend and the KServe v2 control plane (model repository + load/unload). Self-hosted; the provider drives both.

Provider Name	`triton`
Module	`shared.plugins.model_provider.triton`
SDK	`openai` (chat) + `httpx` (KServe v2 control)
Auth	Optional bearer token (Triton has no native auth — sent on both surfaces for fronting proxies)
Mode	Passive — the operator launches Triton and manages its model repository

Highlights

Dual surface — OpenAI chat frontend (default port 9000) + KServe v2 control plane (default port 8000)
Model repository — connect() validates the model against POST /v2/repository/index
Optional load — a load body triggers POST /v2/repository/models/<name>/load at connect; omit it for passive mode
Self-hosted — no API costs, data never leaves your hardware
Function calling — via the OpenAI-compatible tool-calling API

Context Length Is Required

Triton's model config carries no standard context-length field (it's backend-specific — TRT-LLM max_seq_len, vLLM max_model_len, …), so the framework can't auto-discover it. You must set a context length (plugin_configs.triton.context_length, or its env-var equivalent — see jaato-scaffold explain provider triton). The provider raises if unset.

Passive Provider

The provider does not start Triton or build the model repository. Launching tritonserver + its OpenAI frontend and populating the repository live at the deployment boundary. The provider only validates and, when given a load body, loads.

Discover config + verify

# Triton needs both URLs (chat + control) and a context length — the live
# registry is the source of truth, so list them from there rather than here:
jaato-scaffold explain provider triton        # typed knobs (URLs, context, load)
jaato-doctor --workspace . --env-file .env    # confirm both surfaces resolve

Python quick start

from jaato import JaatoClient

client = JaatoClient(provider_name="triton")
client.connect(
    project=None,
    location=None,
    model="llama-3.1-8b"
)
client.configure_tools(registry)

response = client.send_message(
    "Hello from Triton!",
    on_output=on_output
)

Configuration

jaato doesn't duplicate Triton's environment variables here — they drift. Run jaato-scaffold explain provider triton for the full typed list (the two surface URLs, context length, bearer token), and jaato-doctor to verify your environment resolves them. The profile knobs below are the in-YAML equivalents.

Profile Knobs (`plugin_configs.triton`)

Key	Type	Description
`host`	str	Host shorthand (canonical ports)
`openai_url` / `control_url`	str	Explicit per-surface URL overrides
`context_length`	int	Context window (required)
`api_token`	str	Bearer token override
`load`	dict	Body passed to `/v2/repository/models/<name>/load` at connect. Omit (`None`) for passive mode.

Inspect config + verify

jaato-scaffold explain provider triton        # typed knobs + env, from the live registry
jaato-doctor --workspace . --env-file .env

Profile (YAML, with load)

name: triton-llama31
model: llama-3.1-8b
provider: triton

plugins:
  - cli
  - file_edit

plugin_configs:
  triton:
    host: "http://gpu-server"
    context_length: 131072
    # Optional: load the model at connect (omit for passive mode)
    load: {}

Model Repository & Loading

Triton serves from a model repository — a directory of model configs Triton tracks. The provider talks to the KServe v2 control plane to validate and (optionally) load:

connect() validates the requested model against POST /v2/repository/index (a top-level array, distinct from OpenAI's /v1/models shape).
When a load body is supplied, POST /v2/repository/models/<name>/load runs at connect (600s timeout for engine deserialization + KV-cache init). Triton's load is idempotent at the protocol level, so — unlike LM Studio — there is no pre-query for a matching instance.
The connectivity probe targets the control surface (GET /v2/health/live), which load/unload depend on.

The two surfaces

Surface	Default port	Used for
OpenAI chat frontend	`9000`	`POST /v1/chat/completions` (generation)
KServe v2 control plane	`8000`	repository index, load/unload, `/v2/health/live`

Verify both surfaces

# Control-plane liveness (the provider's probe)
curl http://localhost:8000/v2/health/live

# Repository index — the model list connect() validates against
curl -X POST http://localhost:8000/v2/repository/index -d '{}'

# Chat frontend (generation)
curl http://localhost:9000/v1/models

Load body passthrough

plugin_configs:
  triton:
    # Forwarded verbatim to /v2/repository/models/<name>/load.
    # {} = "load with the stored config".
    load: {}

Error Handling

Exception	Cause
`TritonConnectionError`	A surface is unreachable (carries `surface="openai"` for runtime, `surface="control"` for the probe / load)
`TritonAuthenticationError`	Bearer token rejected (HTTP 401)
`TritonModelNotFoundError`	Model not in the repository index
`TritonLoadError`	`/load` returned a non-2xx (carries status + body + the load config)
`TritonMidStreamError`	Connection dropped mid-response (engine error after HTTP 200 committed — check Triton logs)
`ValueError` (at initialize)	OpenAI/control URL or context length not set (no hardcoded fallback)

Handle errors

from shared.plugins.model_provider.triton.errors import (
    TritonConnectionError,
    TritonLoadError,
    TritonMidStreamError,
)

try:
    provider.initialize(config)
    provider.connect("llama-3.1-8b")
except ValueError as e:
    # Missing URLs or context length
    print(f"Configuration error: {e}")
except TritonConnectionError as e:
    print(f"Triton unreachable on the {e.surface} surface")
except TritonLoadError:
    print("Model load failed — check the repository config")
except TritonMidStreamError:
    print("Engine dropped mid-response. Check Triton logs.")