Connection Recovery
Build resilient clients that automatically reconnect to the jaato server after interruptions. Handle server restarts, crashes, and network issues without losing conversation state.
Why Connection Recovery?
When your client connects to the jaato server via IPC, the connection can be interrupted by:
- Server restarts (updates, configuration changes)
- Server crashes (out-of-memory, unhandled errors)
- Network interrupts (for WebSocket clients)
- System hibernation or sleep
Without recovery, your client would need manual reconnection and would
lose track of the active session. IPCRecoveryClient handles
all of this automatically.
Quick Start
Swap IPCClient for IPCRecoveryClient to get
automatic reconnection — the same
.session(...) +
ask/complete/stream facade,
now resilient across daemon restarts. Pass
on_status_change to observe the connection state.
Key Differences from IPCClient
- Reconnection and session reattachment happen automatically after a drop
- Operations raise
ReconnectingErrorduring recovery instead of crashing - The facade dispatches events for you — no manual
events()loop is needed forask/complete/stream(drop to the low-levelevents()API, shown further down, only for custom event handling)
reattach_session: true
(the default). After reconnecting, the client sends a session.attach
command with the stored session ID, and the server restores the conversation state.
import asyncio
from jaato_sdk import IPCRecoveryClient
async def main():
# Same facade as IPCClient.session, on the auto-reconnect client.
# Connect, session creation, and reattachment are handled for you;
# ask() survives a daemon restart. Credentials resolve from the
# profile (e.g. a pass:// knob), never an env var.
async with IPCRecoveryClient.session(
profile={"model": "...", "provider": "...", "plugins": []},
on_status_change=lambda st: print(st.state), # RECONNECTING / CONNECTED
) as s:
print(await s.ask("Long task..."))
asyncio.run(main())
connect() / create_session() /
async for event in client.events()); those methods are
documented in the sections below and on the
IPCRecoveryClient
API page. With the facade, iterating events() yourself
is optional.
Connection States
The recovery client tracks its connection through six states. Understanding these helps you build appropriate UI feedback.
| State | Meaning |
|---|---|
DISCONNECTED |
Initial state, or after a graceful disconnect() |
CONNECTING |
Attempting initial connection or a retry attempt |
CONNECTED |
Active connection, events flowing normally |
RECONNECTING |
Connection lost, waiting for next retry (backoff) |
DISCONNECTING |
Graceful disconnect initiated by client |
CLOSED |
Terminal state — max reconnection attempts exhausted, a permanent error occurred, or close() was called; no more connection attempts |
What Your Client Should Do
- CONNECTING — Show "Connecting..." indicator
- CONNECTED — Normal operation, enable input
- RECONNECTING — Show retry status, disable send
- CLOSED — Show "Disconnected", offer manual reconnect
from jaato_sdk.client import (
IPCRecoveryClient,
ConnectionState,
)
from jaato_sdk.events import ClientType
client = IPCRecoveryClient(
"/tmp/jaato.sock",
client_type=ClientType.TERMINAL,
)
# Check state directly
if client.state == ConnectionState.CONNECTED:
await client.send_message("Hello!")
# Convenience properties
client.is_connected # True when CONNECTED
client.is_reconnecting # True when RECONNECTING
client.is_closed # True when CLOSED
from jaato_sdk.client import ConnectionState
ConnectionState.DISCONNECTED # "disconnected"
ConnectionState.CONNECTING # "connecting"
ConnectionState.CONNECTED # "connected"
ConnectionState.RECONNECTING # "reconnecting"
ConnectionState.DISCONNECTING # "disconnecting"
ConnectionState.CLOSED # "closed"
Status Callback
The on_status_change callback fires on every state transition.
It receives a ConnectionStatus object with the current state
and recovery progress.
ConnectionStatus Fields
| Field | Type | Description |
|---|---|---|
state |
ConnectionState |
Current connection state |
attempt |
int |
Current reconnection attempt (0 when connected) |
max_attempts |
int |
Maximum attempts before giving up |
next_retry_in |
float | None |
Seconds until next retry (during RECONNECTING) |
last_error |
str | None |
Description of the last connection error |
session_id |
str | None |
Active session ID (if any) |
client_id |
str | None |
Client identifier assigned by server |
from jaato_sdk.client import (
IPCRecoveryClient,
ConnectionState,
)
from jaato_sdk.client.recovery import ConnectionStatus
from jaato_sdk.events import ClientType
def on_status(status: ConnectionStatus):
"""Update UI based on connection state."""
if status.state == ConnectionState.CONNECTED:
show_status("Connected")
enable_input()
elif status.state == ConnectionState.RECONNECTING:
msg = (
f"Reconnecting (attempt {status.attempt}"
f"/{status.max_attempts})"
)
if status.next_retry_in is not None:
msg += f" — retry in {status.next_retry_in:.1f}s"
show_status(msg)
disable_input()
elif status.state == ConnectionState.CONNECTING:
show_status("Connecting...")
elif status.state == ConnectionState.CLOSED:
if status.last_error:
show_status(f"Disconnected: {status.last_error}")
else:
show_status("Disconnected")
disable_input()
client = IPCRecoveryClient(
socket_path="/tmp/jaato.sock",
client_type=ClientType.TERMINAL, # required
on_status_change=on_status,
)
Handling Operations During Recovery
When the connection is down, operations like send_message()
raise specific exceptions. Your client should handle these to provide
a good user experience.
Exception Types
| Exception | Classification | When | Action |
|---|---|---|---|
ReconnectingError |
Transient | Client is currently reconnecting | Queue and retry after reconnect |
ConnectionClosedError |
Permanent | Connection permanently closed (max attempts exhausted or close() called) |
Inform user, offer manual restart |
IncompatibleServerError |
Permanent | Server wire-protocol version is below the client's minimum; raised during connect(); client transitions immediately to CLOSED |
Display upgrade message; upgrading one side is the only fix — no retry will help |
close() transitions to the CLOSED state permanently —
no more reconnection attempts. Use disconnect() for temporary
disconnection.
from jaato_sdk.client.recovery import (
ReconnectingError,
ConnectionClosedError,
)
async def send_with_retry(client, message):
"""Send a message, queuing if reconnecting."""
while True:
try:
await client.send_message(message)
return
except ReconnectingError:
# Wait for reconnection, then retry
print("Reconnecting — message queued...")
await wait_for_connected(client)
except ConnectionClosedError:
print("Connection closed permanently.")
raise
async def wait_for_connected(client):
"""Wait until client reconnects."""
while client.is_reconnecting:
await asyncio.sleep(0.5)
if client.is_closed:
raise ConnectionClosedError()
# The events() iterator automatically reconnects.
# It yields events from the new connection
# seamlessly — no special handling needed.
async for event in client.events():
# This keeps working across reconnections.
# You don't need try/except here.
handle_event(event)
Session Reattachment
After reconnecting, the client can reattach to its previous session.
The server loads the session from disk (if evicted from memory) and
sends a SessionInfoEvent with the full session state.
What's Preserved
- Session ID
- Conversation history (persisted on server disk)
- Tool states (managed by server)
What's Lost
- Active IPC connection (replaced by new one)
- In-flight requests (pending permission responses)
- Real-time event stream (restarted after reconnect)
set_session_id() after creating a session —
without it, the recovery client can't reattach after reconnection.
from jaato_sdk.events import ClientType
client = IPCRecoveryClient(
"/tmp/jaato.sock",
client_type=ClientType.TERMINAL,
)
await client.connect()
# Create session and track it
# (optionally with an agent profile)
session_id = await client.create_session(
"work", profile="researcher-claude"
)
client.set_session_id(session_id)
# After reconnection, the client automatically
# sends session.attach with this session_id.
# The server restores conversation history.
# If you have a session ID from a previous run
# (e.g., stored in a config file), you can
# reattach manually:
client = IPCRecoveryClient(
"/tmp/jaato.sock",
client_type=ClientType.TERMINAL,
)
await client.connect()
# Attach to existing session
success = await client.attach_session(
"previous-session-id"
)
if success:
# Track it for future reconnections
client.set_session_id("previous-session-id")
Configuration
Recovery behavior is configured via RecoveryConfig.
Configuration is loaded and merged in precedence order:
- Built-in defaults (lowest)
- User config (
~/.jaato/client.json) - Project config (
.jaato/client.json) - Environment variables (highest)
RecoveryConfig Fields
| Field | Default | Description |
|---|---|---|
enabled |
true |
Enable automatic reconnection |
max_attempts |
10 |
Max reconnection attempts |
base_delay |
1.0 |
Initial backoff delay (seconds) |
max_delay |
60.0 |
Maximum backoff delay cap |
jitter_factor |
0.3 |
Random jitter range (0.3 = ±30%) |
connection_timeout |
5.0 |
Timeout per connection attempt |
reattach_session |
true |
Auto-reattach to previous session |
Environment Variables
| Variable | Config Field |
|---|---|
JAATO_IPC_AUTO_RECONNECT |
enabled |
JAATO_IPC_RETRY_MAX_ATTEMPTS |
max_attempts |
JAATO_IPC_RETRY_BASE_DELAY |
base_delay |
JAATO_IPC_RETRY_MAX_DELAY |
max_delay |
JAATO_IPC_RETRY_JITTER |
jitter_factor |
JAATO_IPC_CONNECTION_TIMEOUT |
connection_timeout |
JAATO_IPC_REATTACH_SESSION |
reattach_session |
{
"recovery": {
"enabled": true,
"max_attempts": 10,
"base_delay": 1.0,
"max_delay": 60.0,
"jitter_factor": 0.3,
"connection_timeout": 5.0,
"reattach_session": true
}
}
from jaato_sdk.client import (
IPCRecoveryClient,
RecoveryConfig,
)
from jaato_sdk.events import ClientType
# Custom config
config = RecoveryConfig(
max_attempts=5,
base_delay=2.0,
max_delay=30.0,
)
client = IPCRecoveryClient(
socket_path="/tmp/jaato.sock",
client_type=ClientType.TERMINAL, # required
config=config,
)
# Or load from config files automatically
from jaato_sdk.client import get_recovery_config
config = get_recovery_config(
workspace_path="/path/to/project"
)
client = IPCRecoveryClient(
socket_path="/tmp/jaato.sock",
client_type=ClientType.TERMINAL, # required
config=config,
)
Python WebSocket Recovery (WSRecoveryClient)
For a remote daemon over WebSocket, the Python SDK
ships WSRecoveryClient — a first-class,
IPCRecoveryClient-equivalent implementation. It is a
subclass of WSClient with the same recovery
machinery as the IPC client: the same six-state
connection state machine, exponential
backoff with jitter, automatic session reattachment, and the
on_status_change callback. You get IPC-grade resilience
over ws:// / wss:// without dropping to raw
frames.
Through the facade, the same
.session(...) +
ask/complete/stream code runs
over WS recovery — either via WSRecoveryClient.session(...)
or the transport-agnostic
jaato.session(mode="ws", recovery=True, ...) flag. The
recovery=True flag works on both daemon transports
(mode="ipc" and mode="ws"); it raises
ValueError for mode="in_process", which has no
daemon to reconnect to.
TLS for self-signed wss://
For a development or internal wss:// endpoint with a
self-signed certificate, pass ssl= (an
ssl.SSLContext, or True/False) or
ca= (a CA-bundle path) — accepted on
WSClient / WSRecoveryClient and on
jaato.session(mode="ws", ca=...). A ca path is
loaded into a default verifying context; ssl wins if both
are set. These are scoped per connection — loaded into
the connection's SSLContext, never into
os.environ — so, unlike an SSL_CERT_FILE env
hack, they cannot leak into a subprocess-restarted daemon's outbound
HTTPS (the Python analog of Node's NODE_EXTRA_CA_CERTS).
jaato-sdk[ws] extrawebsockets package.
WSRecoveryClient imports without it, but
connecting raises ImportError with an install hint. Install
with pip install 'jaato-sdk[ws]'.
import asyncio
import jaato
async def main():
# Same facade as IPCRecoveryClient, over WebSocket. Reconnection,
# session creation, and reattachment are handled for you; ask()
# survives a daemon restart or a dropped socket.
async with jaato.session(
mode="ws",
url="wss://host:8080",
token="...", # bearer token (or ?token= in the url)
recovery=True, # → WSRecoveryClient
ca="/etc/jaato/dev-ca.pem", # trust a self-signed wss:// cert
on_status_change=lambda st: print(st.state), # RECONNECTING / CONNECTED
) as s:
print(await s.ask("Long task..."))
asyncio.run(main())
from jaato_sdk import WSRecoveryClient
from jaato_sdk.events import ClientType
client = WSRecoveryClient(
url="wss://host:8080",
token="...",
client_type=ClientType.API,
ca="/etc/jaato/dev-ca.pem", # or ssl=<ssl.SSLContext>
on_status_change=lambda st: print(st.state),
)
# Same connection-lifecycle API as IPCRecoveryClient:
# connect() / create_session() / send_message() / events().
Browser & Raw-Frame Clients (WebSocket)
Python clients should reach for WSRecoveryClient above.
This section documents the underlying raw WS frame
protocol for browsers / JavaScript and other non-Python clients (the
TypeScript @jaato/sdk recovery: {} wraps these
frames for you). The same recovery concepts apply, but reattachment has
one extra step that the IPC transport does not. Each WebSocket
connection is given its own workspace, so a new connection
cannot see a session created on a previous connection until it selects
that session's workspace first. The reconnect sequence is
session.list → workspace.select →
session.attach.
Key Patterns
- Use a reconnecting WebSocket library (or your own backoff loop)
- Wait for the unprompted
{"type":"connected"}greeting before sending any command - Track the session ID and its
workspace_path(read fromsession.list) client-side - Reattach with
session.list→workspace.select→session.attach—session.attachis acommand.executewith the session ID as the first positional argument - The cold restore is asynchronous: a send issued while it settles can
return a recoverable
SessionError("Session not found") — reattach and resend until output arrives - Handle the ping/pong keepalive mechanism
Authorization: Bearer <token>.
Browsers cannot set custom headers in new WebSocket(),
so pass it as a query parameter instead:
?token=<token>.
Omitting or sending an invalid token causes the server to close the
connection immediately with WS close code 1008
(Policy Violation). The default token path is
$HOME/.jaato/ws.token; it is auto-generated on first server
start if not provided via --ws-token /
--ws-token-file.
"Session not found" rather than naming where the
session lives. This is deliberate: workspace isolation is a security
boundary, and the server will not confirm that a session exists in a
workspace the connection has not selected — otherwise the error could
be used to enumerate sessions across workspaces. workspace.select
is the legitimate way to re-target a workspace you own.
// Raw WebSocket reconnect (no SDK). With @jaato/sdk, `recovery: {}` does all of
// this for you; this shows the underlying frames.
class JaatoWebClient {
private ws: WebSocket | null = null;
private sessionId: string | null = null;
private workspacePath: string | null = null; // the session's own workspace
private reconnectAttempt = 0;
private maxAttempts = 10;
// url carries the bearer token: wss://<host>/?token=<token>
// For a self-signed dev wss://, trust the dev CA out-of-band
// (e.g. NODE_EXTRA_CA_CERTS), never by disabling verification.
connect(url: string) {
this.ws = new WebSocket(url);
this.ws.onmessage = (e) => {
const frame = JSON.parse(e.data);
switch (frame.type) {
case "connected": // unprompted greeting — only now is it safe to send
this.reconnectAttempt = 0;
if (this.sessionId) this.reattach();
break;
case "session.list": // find the session's workspace, then select + attach
this.workspacePath =
frame.sessions.find((s) => s.id === this.sessionId)?.workspace_path;
this.send({ type: "workspace.select", name: this.workspacePath });
this.send({ type: "command.execute",
command: "session.attach", args: [this.sessionId] });
break;
case "error": // recoverable "Session not found" while the ~15s cold
if (frame.recoverable) this.reattach(); // restore settles — resend
break;
}
};
this.ws.onclose = () => this.scheduleReconnect(url);
}
// A fresh connection gets its own workspace, so the session created on an
// earlier connection isn't reachable until we select its workspace. List
// sessions to find its workspace_path; the handler above does select + attach.
private reattach() {
this.send({ type: "command.execute", command: "session.list", args: [] });
}
private scheduleReconnect(url: string) {
if (this.reconnectAttempt >= this.maxAttempts) return; // give up
const delay = Math.min(60_000, 1000 * Math.pow(2, this.reconnectAttempt));
this.reconnectAttempt++;
setTimeout(() => this.connect(url), delay);
}
private send(frame: unknown) {
this.ws?.send(JSON.stringify(frame));
}
}
Next Steps
- IPCRecoveryClient API Reference — Full method signatures, parameters, and return types
- IPC Recovery Architecture — Deep dive into backoff algorithms, error classification, and state machine design
- Client — Core client concepts and the
JaatoClientAPI - JaatoClient API Reference — Full API documentation
# Imports
from jaato_sdk.client import (
IPCRecoveryClient,
ConnectionState,
RecoveryConfig,
get_recovery_config,
)
from jaato_sdk.client.recovery import (
ConnectionStatus,
ReconnectingError,
ConnectionClosedError,
IncompatibleServerError,
)
from jaato_sdk.events import ClientType
# States
ConnectionState.DISCONNECTED
ConnectionState.CONNECTING
ConnectionState.CONNECTED
ConnectionState.RECONNECTING
ConnectionState.DISCONNECTING
ConnectionState.CLOSED
# Key methods
client.connect() # Start connection
client.disconnect() # Temporary disconnect
client.close() # Permanent close
client.set_session_id(id) # Track for recovery
client.send_message(text) # Send (raises on down)
client.events() # Auto-reconnecting stream
# Properties
client.state # ConnectionState
client.is_connected # bool
client.is_reconnecting # bool
client.is_closed # bool
client.session_id # str | None