Provider Reference

Model providers connect jaato to AI services. Each provider handles authentication, API communication, and type conversion for a specific AI platform.

Available Providers

Provider	Name	Status	Models
Google GenAI	`google_genai`	Available	Gemini 1.5, 2.0, 2.5
Anthropic	`anthropic`	Available	Claude 3, 3.5, Sonnet 4, Opus 4
GitHub Models	`github_models`	Available	GPT-4o, Claude 3.5, Gemini, Llama
Claude CLI	`claude_cli`	Available	Claude (via subscription)
Antigravity	`antigravity`	Available	Gemini 3, Claude (via Google)
Ollama	`ollama`	Available	Qwen, Llama, Mistral (local)
Zhipu AI	`zhipuai`	Available	GLM-5, GLM-4.7, GLM-4.6, GLM-4.5
NVIDIA NIM	`nim`	Available	Llama, DeepSeek-R1, Nemotron
LM Studio	`lmstudio`	Available	Any model loaded in LM Studio (local)
vLLM	`vllm`	Available	200+ architectures via self-hosted vLLM server (local/GPU)
TensorRT-LLM	`tensorrt_llm`	Available	NVIDIA TensorRT-LLM engines via trtllm-serve (self-hosted GPU)
Triton	`triton`	Available	Triton Inference Server — OpenAI frontend + KServe v2 (self-hosted)
OpenRouter	`openrouter`	Available	300+ models (OpenAI, Anthropic, Google, Meta, Mistral, DeepSeek…)
Nebius Token Factory	`nebius`	Available	Serverless open models (Llama, Qwen, DeepSeek-R1, Mistral…)
OVHcloud AI Endpoints	`ovhcloud`	Available	Serverless open models on OVHcloud's EU cloud (Llama, Mistral, Qwen, gpt-oss…)

Choosing a Provider

Use Case	Recommended
Personal development	Google GenAI (AI Studio) - free tier available
Claude Pro/Max subscriber	Claude CLI - uses subscription, not API credits
Complex reasoning tasks	Anthropic - Claude with extended thinking
Multi-model access	GitHub Models - GPT, Claude, Gemini via single API
Privacy / Local models	Ollama - run models locally, no API costs
GCP integration	Google GenAI (Vertex AI)
Access to Gemini 3	Antigravity - Google IDE backend
GLM models / China AI	Zhipu AI - GLM-5, GLM-4.7 with chain-of-thought
NVIDIA GPU infrastructure	NVIDIA NIM - hosted or self-hosted, Llama/DeepSeek-R1/Nemotron
Local GUI model management	LM Studio - GUI, load control, any HuggingFace model
High-throughput GPU inference (self-hosted)	vLLM - PagedAttention, 200+ architectures, LoRA hot-loading
Maximum GPU throughput (custom engine)	TensorRT-LLM - build your own FP8/INT4 engine with trtllm-build
Multi-vendor / single API key	OpenRouter - 300+ models, cross-model fallback, provider routing

Using a provider

from jaato import JaatoClient

# Recommended: configure via .env file
# JAATO_PROVIDER=anthropic
# MODEL_NAME=claude-sonnet-4-20250514
client = JaatoClient()
client.connect()  # Reads provider and model from env

# Override provider in code (model still from env)
client = JaatoClient(provider_name="anthropic")
client.connect()

# Override both provider and model in code
client = JaatoClient(provider_name="google_genai")
client.connect(model="gemini-2.5-flash")

client = JaatoClient(provider_name="ollama")
client.connect(model="qwen3:32b")

Discover available providers

from shared import discover_providers, load_provider

# Find what's available
providers = discover_providers()
print(f"Available: {list(providers.keys())}")
# ['google_genai', 'anthropic', 'github_models',
#  'claude_cli', 'antigravity', 'ollama', 'zhipuai', 'nim',
#  'lmstudio', 'vllm', 'tensorrt_llm', 'openrouter']

# Load a specific provider
provider = load_provider("anthropic")
print(f"Loaded: {provider.name}")

Provider Selection

There are three ways to select which provider to use, in order of precedence:

1. CLI Flag (highest priority)

Use the --provider flag when running the rich client. This overrides all other settings.

2. Environment Variable

Set JAATO_PROVIDER in your .env file or shell. This is the recommended approach for persistent configuration.

3. Code (explicit)

Pass provider_name to JaatoClient(). Overrides environment variable.

4. Default

If nothing is specified, defaults to google_genai. Set JAATO_PROVIDER to change the default.

Variable	Values	Description
`JAATO_PROVIDER`	`anthropic`, `google_genai`, `github_models`, `claude_cli`, `antigravity`, `ollama`, `zhipuai`, `nim`, `lmstudio`, `vllm`, `tensorrt_llm`, `openrouter`	Default provider for all sessions

CLI flag (overrides env)

# Use GitHub Models for this session
.venv/bin/python jaato-tui/rich_client.py --provider github_models

# Use Google GenAI for this session
.venv/bin/python jaato-tui/rich_client.py --provider google_genai

Environment variable (.env)

# .env file
JAATO_PROVIDER=github_models
MODEL_NAME=openai/gpt-4o

Explicit in code

from jaato import JaatoClient

# Explicit provider (overrides JAATO_PROVIDER env var)
client = JaatoClient(provider_name="anthropic")
client.connect()  # Reads MODEL_NAME from env

# Everything from env
client = JaatoClient()
client.connect()  # Reads JAATO_PROVIDER and MODEL_NAME from env

Provider Comparison

Google GenAI

Endpoints: AI Studio (personal) + Vertex AI (enterprise)
Auth: API key, ADC, service account
Features: Function calling, multimodal, structured output
Best for: GCP users, long context needs (1M+ tokens)

Anthropic

Endpoints: Direct API
Auth: API key (sk-ant-api03-...) or OAuth token (sk-ant-oat01-...)
Features: Function calling, extended thinking, prompt caching
Best for: Complex reasoning, code generation

GitHub Models

Endpoints: GitHub Models API (unified)
Auth: GitHub PAT (fine-grained recommended)
Features: Multi-model (GPT, Claude, Gemini, Llama), function calling
Best for: GitHub Enterprise users, multi-model experimentation

Claude CLI

Endpoints: Claude Code CLI wrapper
Auth: claude login (uses Pro/Max subscription)
Features: Built-in tools, automatic prompt caching
Best for: Claude subscribers wanting to avoid API costs

Antigravity

Endpoints: Google IDE backend
Auth: Google OAuth (via oauth_login())
Features: Access to Gemini 3, Claude via Google infrastructure
Best for: Early access to Gemini 3 models

Ollama

Endpoints: Local Ollama server (localhost:11434)
Auth: None (local)
Features: Run models locally, no API costs, privacy
Best for: Privacy-sensitive use cases, offline work

Zhipu AI

Endpoints: Z.AI Anthropic-compatible API
Auth: API key
Features: GLM models, extended thinking, function calling
Best for: GLM model access, chain-of-thought reasoning

NVIDIA NIM

Endpoints: NVIDIA hosted API + self-hosted NIM containers
Auth: API key (nvapi-...) or none (self-hosted)
Features: OpenAI-compatible API, function calling, streaming, reasoning (DeepSeek-R1)
Best for: NVIDIA GPU infrastructure, self-hosted models, NIM catalog access

LM Studio

Endpoints: Local LM Studio server (localhost:1234)
Auth: None (local) or optional bearer token
Features: GUI model management, load-control API, context length auto-detection, function calling
Best for: Local development with GUI, precise per-session GPU/context configuration

vLLM

Endpoints: Self-hosted vLLM OpenAI-compatible server
Auth: Optional bearer token (--api-key at server launch)
Features: PagedAttention, continuous batching, 200+ model architectures, LoRA hot-loading, quirks system for small models
Best for: High-throughput self-hosted GPU inference, broad model architecture support

TensorRT-LLM

Endpoints: Self-hosted trtllm-serve instance
Auth: Optional bearer token (auth proxy only; trtllm-serve has no built-in key mechanism)
Features: FP8/INT4 quantization, in-flight batching, KV-cache reuse, speculative decoding
Best for: Maximum GPU throughput, custom-built TensorRT-LLM engines, NVIDIA H100/A100/L40S

OpenRouter

Endpoints: OpenRouter API (https://openrouter.ai/api/v1)
Auth: API key (sk-or-...)
Features: 300+ models, cross-model fallback, provider routing, prompt caching, extended thinking, openrouter/auto
Best for: Multi-provider access via a single key, fallback resilience, model experimentation

Provider capabilities

# Check provider capabilities
provider = load_provider("anthropic")  # or any provider

# Structured output support
if provider.supports_structured_output():
    response = provider.send_message(
        "Extract data",
        response_schema={"type": "object", ...}
    )

# Context limits
limit = provider.get_context_limit()
print(f"Context window: {limit:,} tokens")

# Token counting
tokens = provider.count_tokens("Hello world")
print(f"Token count: {tokens}")

Next Steps

Anthropic Provider - Claude API, OAuth, extended thinking
Google GenAI Provider - GCP integration, Vertex AI
GitHub Models Provider - Multi-model access, enterprise SSO
Ollama Provider - Local models, no API costs
Zhipu AI Provider - GLM models, chain-of-thought reasoning
NVIDIA NIM Provider - Hosted and self-hosted NIM models
LM Studio Provider - Local GUI, load control
vLLM Provider - High-throughput GPU inference, quirks system
TensorRT-LLM Provider - Custom TensorRT-LLM engines
OpenRouter Provider - 300+ models, provider routing
Provider Concepts - How providers work
Types Reference - Provider-agnostic types

Provider directory structure

shared/plugins/model_provider/
├── __init__.py
├── base.py              # Protocol definition
├── types.py             # Unified types
├── anthropic/           # Anthropic Claude
├── google_genai/        # Google GenAI / Vertex AI
├── github_models/       # GitHub Models API
├── claude_cli/          # Claude CLI wrapper
├── antigravity/         # Google IDE backend
├── ollama/              # Ollama local models
├── zhipuai/             # Zhipu AI GLM models
├── nim/                 # NVIDIA NIM (hosted + self-hosted)
├── lmstudio/            # LM Studio local models (OpenAI-compat + load control)
├── vllm/                # vLLM (OpenAI-compat, self-hosted GPU, quirks)
├── tensorrt_llm/        # TensorRT-LLM via trtllm-serve (self-hosted GPU)
└── openrouter/          # OpenRouter (300+ models, unified gateway)