vLLM Provider
High-throughput self-hosted GPU inference via vLLM's OpenAI-compatible server. Supports 200+ model architectures, PagedAttention for memory efficiency, continuous batching, LoRA adapter hot-loading, and prefix caching.
| Provider Name | vllm |
| Module | shared.plugins.model_provider.vllm |
| SDK | openai (OpenAI-compatible API) |
| Auth | Optional bearer token (only when --api-key passed to vllm serve) |
| Mode | Passive — provider talks to an already-running vLLM server |
Highlights
- High throughput — PagedAttention, continuous batching, prefix caching
- 200+ architectures — Qwen, Llama, Mistral, DeepSeek, Phi, Gemma, and more
- LoRA adapters — Hot-loading of fine-tuned adapters
- Structured outputs — xgrammar-constrained decoding via
--enable-guided-decoding - Tool calling — Via
--enable-auto-tool-choice --tool-call-parserat server launch - Quirks system — Profile-level workarounds for small-model behavior gaps
Passive Provider — No Load Endpoint
The provider does not start or configure vLLM. Model choice
(--model), context length (--max-model-len), and
tool-call parser (--enable-auto-tool-choice --tool-call-parser)
all live at the vllm serve command boundary. Configure them once
at server launch, not per-session.
Start vLLM server
vllm serve Qwen/Qwen2.5-7B-Instruct \
--host 0.0.0.0 --port 8000 \
--max-model-len 32768 \
--enable-auto-tool-choice \
--tool-call-parser hermesConnect jaato
export VLLM_HOST=http://localhost:8000
export VLLM_MODEL=Qwen/Qwen2.5-7B-Instruct
export VLLM_CONTEXT_LENGTH=32768Python quick start
from jaato import JaatoClient
client = JaatoClient(provider_name="vllm")
client.connect(
project=None,
location=None,
model="Qwen/Qwen2.5-7B-Instruct"
)
client.configure_tools(registry)
response = client.send_message(
"Hello from vLLM!",
on_output=on_output
)Configuration
Environment Variables
| Variable | Default | Required | Description |
|---|---|---|---|
VLLM_HOST |
— | Yes | vLLM server URL (e.g. http://localhost:8000). No localhost fallback. |
VLLM_MODEL |
— | Recommended | Model name as vLLM reports it in /v1/models |
VLLM_CONTEXT_LENGTH |
Auto-detected | If auto-detect fails | Context window size. Current vLLM surfaces max_model_len in /v1/models; set this only for older builds or to pin a value. |
VLLM_API_TOKEN |
— | No | Bearer token when server was launched with --api-key or sits behind an auth proxy |
Context Length Auto-Detection
Current vLLM versions surface max_model_len in each
GET /v1/models entry. The provider reads this at
initialize() time (tier-1). VLLM_CONTEXT_LENGTH
is a tier-3 fallback for older builds. If neither resolves, the provider
raises rather than using a hardcoded default.
Profile Knobs (plugin_configs.vllm)
| Key | Type | Description |
|---|---|---|
host |
str | Override VLLM_HOST |
context_length |
int | Override context window (tier-2; skips auto-detect) |
api_token |
str | Bearer token override |
max_tokens |
int | Cap per-request output budget; forwarded as OpenAI max_tokens. Omit to use vLLM's default. |
Full .env example
# .env
JAATO_PROVIDER=vllm
VLLM_HOST=http://localhost:8000
VLLM_MODEL=Qwen/Qwen2.5-7B-Instruct
VLLM_CONTEXT_LENGTH=32768Profile with per-session override
name: vllm-qwen
model: Qwen/Qwen2.5-7B-Instruct
provider: vllm
plugins:
- cli
- file_edit
plugin_configs:
vllm:
host: "http://gpu-host:8000"
context_length: 32768
max_tokens: 4096Tool Calling
Tool calling requires configuring vLLM at server launch with the correct tool-call parser for your model family. The provider passes tool schemas using the OpenAI wire format; the server-side parser extracts tool calls from the model's output.
Tool-Call Parser by Model Family
| Parser Flag | Models |
|---|---|
hermes |
Qwen2.5, Hermes |
mistral |
Mistral Instruct |
llama3_json |
Llama 3.1 |
pythonic |
Llama 3.2, Llama 4 |
granite |
IBM Granite |
deepseek_v4 |
DeepSeek-V4 / Qwen 3.5 |
See the vLLM Tool-Calling docs for the full parser list.
Launch vLLM with tool calling (Qwen2.5)
vllm serve Qwen/Qwen2.5-7B-Instruct \
--host 0.0.0.0 --port 8000 \
--max-model-len 32768 \
--enable-auto-tool-choice \
--tool-call-parser hermesLaunch with Llama 3.1 (llama3_json)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 --port 8000 \
--max-model-len 131072 \
--enable-auto-tool-choice \
--tool-call-parser llama3_jsonQuirks System
vLLM is the only jaato provider that implements a quirks subsystem.
Quirks are profile-level workarounds for behavioral gaps in small models running
on vLLM, activated via profile.quirks.*. All quirks default to
false and are opt-in.
Available Quirks
| Quirk Key | Since | What It Does |
|---|---|---|
coerce_typed_tool_args |
server 0.6.194+ |
When the model emits a string value for a tool argument whose schema type
is array, object, integer,
number, or boolean, the provider attempts
ast.literal_eval (handles Python repr with single quotes)
then json.loads to coerce it before passing to the tool executor.
Workaround for Llama 3.1 on vLLM with the llama3_json parser
under tool_choice: "auto".
|
force_tool_choice_for_lifecycle |
server 0.6.195+ |
Forwards the session's tool_choice kwarg to vLLM as a named-function
constraint ({"type":"function","function":{"name":"..."}}}), engaging
vLLM's xgrammar decoding. Produces correctly-typed args at the source instead of
coercing them after. Effective universally (unlike the parser-tag-gated coerce quirk).
|
force_narration_between_tools |
server 0.6.197+ |
After every tool_result turn, the session injects a synthetic
USER-role prompt asking the model to produce 1–2 sentences of narrative
before the next tool call. Closes the small-model narration-skipping failure
class where models like Qwen3-14b at temperature 0 skip narration entirely
even with in-context examples.
|
auto_finalize_on_complete |
server 0.6.199+ |
When the composite is_complete gate flips true (schema floor met,
no completeness processor reports incomplete[]), the framework
synthesizes signal_completion() server-side without a model
round-trip. Prevents context-overflow-at-finalize: the accumulator model
would otherwise over-run the context window attempting another turn.
|
Profile with quirks (YAML)
name: vllm-llama31-quirks
model: meta-llama/Llama-3.1-8B-Instruct
provider: vllm
plugin_configs:
vllm:
host: "http://localhost:8000"
context_length: 131072
# Opt-in workarounds for Llama 3.1 + hermes parser
quirks:
coerce_typed_tool_args: true
force_tool_choice_for_lifecycle: trueQuirks for small-model cascade workloads
name: vllm-qwen3-14b-cascade
model: Qwen/Qwen3-14B-Instruct
provider: vllm
plugin_configs:
vllm:
host: "http://localhost:8000"
context_length: 32768
quirks:
force_narration_between_tools: true
auto_finalize_on_complete: trueError Handling
| Exception | Cause |
|---|---|
VLLMConnectionError |
Server unreachable (not running, wrong host/port, firewall); transient — reliability layer retries |
VLLMAuthenticationError |
Bearer token rejected (HTTP 401 from /health probe) |
VLLMModelNotFoundError |
Requested model is not in /v1/models (server is hosting a different model) |
VLLMMidStreamError |
Connection dropped mid-response (HTTP 200 committed but engine terminated early — check server logs for OOM / generation errors) |
Mid-Stream vs Pre-Flight Errors
The provider distinguishes mid-stream drops (engine error after HTTP 200 was
committed — check vLLM server logs for the root cause) from pre-flight
connection failures (host unreachable / firewall / DNS). These are surfaced
as different exception types so you can route them to different remediation paths.
Handle errors
from shared.plugins.model_provider.vllm.errors import (
VLLMConnectionError,
VLLMModelNotFoundError,
VLLMMidStreamError,
)
try:
provider.initialize(config)
provider.connect("Qwen/Qwen2.5-7B-Instruct")
result = provider.complete(messages, tools=tools)
except VLLMConnectionError:
print("vLLM server not reachable. Is it running?")
except VLLMModelNotFoundError as e:
print(f"Model not loaded: {e}")
print("Check 'vllm serve' is using the right --model flag")
except VLLMMidStreamError:
print("vLLM dropped mid-response. Check server logs.")Health check
# vLLM's liveness probe — 200 = healthy, 503 = engine dead
curl http://localhost:8000/health
# List served models
curl http://localhost:8000/v1/modelsCancellation Behaviour
When a streaming response is cancelled, the provider closes the OpenAI
client's HTTP connection pool immediately (client.close()),
which sends a TCP FIN to vLLM and stops GPU generation within ~50ms.
Without this, vLLM continues generating tokens until max_tokens
is reached because the OpenAI SDK's Stream.close() releases
the wrapper without aggressively closing the underlying TCP socket, which
httpx keeps alive for keep-alive reuse.
Per-Session Providers
Providers are per-session, so closing the client on cancel only affects
the cancelled session. The next session gets a fresh client.
Cancel a streaming request
from jaato import JaatoClient
client = JaatoClient(provider_name="vllm")
client.connect(None, None, "Qwen/Qwen2.5-7B-Instruct")
# Start streaming in background (framework handles this)
# Call client.stop() to cancel:
client.stop() # Signals cancel_token; provider closes TCP conn