TensorRT-LLM Provider

Maximum-throughput NVIDIA GPU inference via trtllm-serve — the HTTP front-end for TensorRT-LLM engines. Supports FP8/INT4 quantization, in-flight batching, KV-cache reuse, and speculative decoding for NVIDIA H100, A100, and L40S hardware.

Provider Name	`tensorrt_llm`
Module	`shared.plugins.model_provider.tensorrt_llm`
SDK	`openai` (OpenAI-compatible API)
Auth	Optional bearer token (only when fronted by an auth proxy; `trtllm-serve` has no built-in API key mechanism)
Mode	Passive — one engine per `trtllm-serve` process

Highlights

Maximum throughput — FP8/INT4 quantization, in-flight batching, KV-cache reuse
Speculative decoding — Reduce latency with a draft model
DIY NIM — Build your own TensorRT-LLM engine; NIM is essentially TensorRT-LLM productized
Self-hosted — No API costs, data never leaves your hardware
Function calling — Via the OpenAI-compatible tool-calling API

Context Length Is Required

trtllm-serve does not surface max_seq_len in its /v1/models response (it is fixed at engine build time). You must set TENSORRT_LLM_CONTEXT_LENGTH (or plugin_configs.tensorrt_llm.context_length) to match the value you used with trtllm-build. The provider raises if neither is set.

Passive Provider

The provider does not build or load engines. Engine build (trtllm-build) and server configuration (tensor parallelism, KV-cache fraction, max batch size) all live at the trtllm-serve launch boundary. Each trtllm-serve process hosts exactly one engine.

Launch trtllm-serve

# Serve a TensorRT-LLM engine
trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 --port 8000

Connect jaato

export TENSORRT_LLM_HOST=http://localhost:8000
export TENSORRT_LLM_MODEL=meta-llama/Llama-3.1-8B-Instruct
# Required: match the engine's max_seq_len from trtllm-build
export TENSORRT_LLM_CONTEXT_LENGTH=131072

Python quick start

from jaato import JaatoClient

client = JaatoClient(provider_name="tensorrt_llm")
client.connect(
    project=None,
    location=None,
    model="meta-llama/Llama-3.1-8B-Instruct"
)
client.configure_tools(registry)

response = client.send_message(
    "Hello from TensorRT-LLM!",
    on_output=on_output
)

Configuration

Environment Variables

Variable	Default	Required	Description
`TENSORRT_LLM_HOST`	—	Yes	`trtllm-serve` URL (e.g. `http://localhost:8000`). No localhost fallback.
`TENSORRT_LLM_MODEL`	—	Recommended	Model name as it appears in `/v1/models`
`TENSORRT_LLM_CONTEXT_LENGTH`	—	Yes	Context window size. Must match the engine's `max_seq_len` from `trtllm-build`. Provider raises if unset.
`TENSORRT_LLM_API_TOKEN`	—	No	Bearer token (only when fronted by an auth proxy)

Profile Knobs (`plugin_configs.tensorrt_llm`)

Key	Type	Description
`host`	str	Override `TENSORRT_LLM_HOST`
`context_length`	int	Context window override (required for long-context engines)
`api_token`	str	Bearer token override

Full .env example

# .env
JAATO_PROVIDER=tensorrt_llm
TENSORRT_LLM_HOST=http://localhost:8000
TENSORRT_LLM_MODEL=meta-llama/Llama-3.1-8B-Instruct
TENSORRT_LLM_CONTEXT_LENGTH=131072

Profile (YAML)

name: trtllm-llama31
model: meta-llama/Llama-3.1-8B-Instruct
provider: tensorrt_llm

plugins:
  - cli
  - file_edit

plugin_configs:
  tensorrt_llm:
    host: "http://gpu-server:8000"
    context_length: 131072

Engine Build and Serve

TensorRT-LLM engines are built out-of-band with trtllm-build from HuggingFace checkpoints. The engine is then served with trtllm-serve. The context length (--max_seq_len) is fixed at build time and must match TENSORRT_LLM_CONTEXT_LENGTH.

vs NVIDIA NIM

The NVIDIA NIM provider wraps the same TensorRT-LLM runtime in a productized container with automatic engine selection. Use this provider when you need to build and optimize your own engine (custom quantization recipes, private models, GPU-specific tuning).

Feature	TensorRT-LLM (this provider)	NVIDIA NIM
Engine management	You build with `trtllm-build`	NIM manages automatically
Model selection	Any HuggingFace model you convert	NIM catalog only
Quantization	Full control (FP8, INT4, INT8)	NIM-chosen per container
Context length	Must be set manually (env var)	NIM reports it; override optional
Hosted option	No (self-hosted only)	Yes (build.nvidia.com)

Build and serve a TensorRT-LLM engine

# 1. Convert HuggingFace checkpoint
python -m tensorrt_llm.commands.convert_checkpoint \
    --model_dir meta-llama/Llama-3.1-8B-Instruct \
    --output_dir ./llama31-trt-ckpt \
    --dtype bfloat16

# 2. Build engine
trtllm-build \
    --checkpoint_dir ./llama31-trt-ckpt \
    --output_dir ./llama31-engine \
    --max_seq_len 131072 \
    --max_batch_size 32

# 3. Serve
trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 --port 8000

# 4. Connect jaato (context_length MUST match --max_seq_len above)
export TENSORRT_LLM_CONTEXT_LENGTH=131072

Verify server

# Health probe
curl http://localhost:8000/health

# Model listing (note: no max_seq_len in response)
curl http://localhost:8000/v1/models

Error Handling

Exception	Cause
`TensorRTLLMConnectionError`	Server unreachable (not running, wrong host/port)
`TensorRTLLMAuthenticationError`	Bearer token rejected by upstream proxy (HTTP 401)
`TensorRTLLMModelNotFoundError`	Requested model name not in `/v1/models`
`TensorRTLLMMidStreamError`	Connection dropped mid-response (engine error after HTTP 200 committed — check `trtllm-serve` logs)
`ValueError` (at initialize)	`TENSORRT_LLM_HOST` or `TENSORRT_LLM_CONTEXT_LENGTH` not set (no hardcoded fallback)

Handle errors

from shared.plugins.model_provider.tensorrt_llm.errors import (
    TensorRTLLMConnectionError,
    TensorRTLLMMidStreamError,
)

try:
    provider.initialize(config)
    provider.connect("meta-llama/Llama-3.1-8B-Instruct")
except ValueError as e:
    # Missing TENSORRT_LLM_HOST or TENSORRT_LLM_CONTEXT_LENGTH
    print(f"Configuration error: {e}")
except TensorRTLLMConnectionError:
    print("trtllm-serve not reachable")
except TensorRTLLMMidStreamError:
    print("Engine dropped mid-response. Check trtllm-serve logs.")