Connection Recovery

Build resilient clients that automatically reconnect to the jaato server after interruptions. Handle server restarts, crashes, and network issues without losing conversation state.

Why Connection Recovery?

When your client connects to the jaato server via IPC, the connection can be interrupted by:

  • Server restarts (updates, configuration changes)
  • Server crashes (out-of-memory, unhandled errors)
  • Network interrupts (for WebSocket clients)
  • System hibernation or sleep

Without recovery, your client would need manual reconnection and would lose track of the active session. IPCRecoveryClient handles all of this automatically.

State machine overview
DISCONNECTED CONNECTING CONNECTED lost RECONNECTING retry CLOSED max DISCON- NECTING

Quick Start

Swap IPCClient for IPCRecoveryClient to get automatic reconnection — the same .session(...) + ask/complete/stream facade, now resilient across daemon restarts. Pass on_status_change to observe the connection state.

Key Differences from IPCClient

  • Reconnection and session reattachment happen automatically after a drop
  • Operations raise ReconnectingError during recovery instead of crashing
  • The facade dispatches events for you — no manual events() loop is needed for ask/complete/stream (drop to the low-level events() API, shown further down, only for custom event handling)
Session Reattachment
Session reattachment is automatic when reattach_session: true (the default). After reconnecting, the client sends a session.attach command with the stored session ID, and the server restores the conversation state.
Minimal recovery client (facade)
import asyncio
from jaato_sdk import IPCRecoveryClient

async def main():
    # Same facade as IPCClient.session, on the auto-reconnect client.
    # Connect, session creation, and reattachment are handled for you;
    # ask() survives a daemon restart. Credentials resolve from the
    # profile (e.g. a pass:// knob), never an env var.
    async with IPCRecoveryClient.session(
        profile={"model": "...", "provider": "...", "plugins": []},
        on_status_change=lambda st: print(st.state),  # RECONNECTING / CONNECTED
    ) as s:
        print(await s.ask("Long task..."))

asyncio.run(main())
Need the raw event stream?
The facade covers the common case. For custom event handling you can still use the low-level client directly (connect() / create_session() / async for event in client.events()); those methods are documented in the sections below and on the IPCRecoveryClient API page. With the facade, iterating events() yourself is optional.

Connection States

The recovery client tracks its connection through six states. Understanding these helps you build appropriate UI feedback.

State Meaning
DISCONNECTED Initial state, or after a graceful disconnect()
CONNECTING Attempting initial connection or a retry attempt
CONNECTED Active connection, events flowing normally
RECONNECTING Connection lost, waiting for next retry (backoff)
DISCONNECTING Graceful disconnect initiated by client
CLOSED Terminal state — max reconnection attempts exhausted, a permanent error occurred, or close() was called; no more connection attempts

What Your Client Should Do

  • CONNECTING — Show "Connecting..." indicator
  • CONNECTED — Normal operation, enable input
  • RECONNECTING — Show retry status, disable send
  • CLOSED — Show "Disconnected", offer manual reconnect
Checking state
from jaato_sdk.client import (
    IPCRecoveryClient,
    ConnectionState,
)
from jaato_sdk.events import ClientType

client = IPCRecoveryClient(
    "/tmp/jaato.sock",
    client_type=ClientType.TERMINAL,
)

# Check state directly
if client.state == ConnectionState.CONNECTED:
    await client.send_message("Hello!")

# Convenience properties
client.is_connected      # True when CONNECTED
client.is_reconnecting   # True when RECONNECTING
client.is_closed         # True when CLOSED
ConnectionState enum values
from jaato_sdk.client import ConnectionState

ConnectionState.DISCONNECTED   # "disconnected"
ConnectionState.CONNECTING     # "connecting"
ConnectionState.CONNECTED      # "connected"
ConnectionState.RECONNECTING   # "reconnecting"
ConnectionState.DISCONNECTING  # "disconnecting"
ConnectionState.CLOSED         # "closed"

Status Callback

The on_status_change callback fires on every state transition. It receives a ConnectionStatus object with the current state and recovery progress.

ConnectionStatus Fields

Field Type Description
state ConnectionState Current connection state
attempt int Current reconnection attempt (0 when connected)
max_attempts int Maximum attempts before giving up
next_retry_in float | None Seconds until next retry (during RECONNECTING)
last_error str | None Description of the last connection error
session_id str | None Active session ID (if any)
client_id str | None Client identifier assigned by server
Status callback implementation
from jaato_sdk.client import (
    IPCRecoveryClient,
    ConnectionState,
)
from jaato_sdk.client.recovery import ConnectionStatus
from jaato_sdk.events import ClientType

def on_status(status: ConnectionStatus):
    """Update UI based on connection state."""
    if status.state == ConnectionState.CONNECTED:
        show_status("Connected")
        enable_input()

    elif status.state == ConnectionState.RECONNECTING:
        msg = (
            f"Reconnecting (attempt {status.attempt}"
            f"/{status.max_attempts})"
        )
        if status.next_retry_in is not None:
            msg += f" — retry in {status.next_retry_in:.1f}s"
        show_status(msg)
        disable_input()

    elif status.state == ConnectionState.CONNECTING:
        show_status("Connecting...")

    elif status.state == ConnectionState.CLOSED:
        if status.last_error:
            show_status(f"Disconnected: {status.last_error}")
        else:
            show_status("Disconnected")
        disable_input()

client = IPCRecoveryClient(
    socket_path="/tmp/jaato.sock",
    client_type=ClientType.TERMINAL,   # required
    on_status_change=on_status,
)

Handling Operations During Recovery

When the connection is down, operations like send_message() raise specific exceptions. Your client should handle these to provide a good user experience.

Exception Types

Exception Classification When Action
ReconnectingError Transient Client is currently reconnecting Queue and retry after reconnect
ConnectionClosedError Permanent Connection permanently closed (max attempts exhausted or close() called) Inform user, offer manual restart
IncompatibleServerError Permanent Server wire-protocol version is below the client's minimum; raised during connect(); client transitions immediately to CLOSED Display upgrade message; upgrading one side is the only fix — no retry will help
close() Is Permanent
close() transitions to the CLOSED state permanently — no more reconnection attempts. Use disconnect() for temporary disconnection.
Queue-and-retry pattern
from jaato_sdk.client.recovery import (
    ReconnectingError,
    ConnectionClosedError,
)

async def send_with_retry(client, message):
    """Send a message, queuing if reconnecting."""
    while True:
        try:
            await client.send_message(message)
            return
        except ReconnectingError:
            # Wait for reconnection, then retry
            print("Reconnecting — message queued...")
            await wait_for_connected(client)
        except ConnectionClosedError:
            print("Connection closed permanently.")
            raise

async def wait_for_connected(client):
    """Wait until client reconnects."""
    while client.is_reconnecting:
        await asyncio.sleep(0.5)
    if client.is_closed:
        raise ConnectionClosedError()
events() handles reconnection automatically
# The events() iterator automatically reconnects.
# It yields events from the new connection
# seamlessly — no special handling needed.

async for event in client.events():
    # This keeps working across reconnections.
    # You don't need try/except here.
    handle_event(event)

Session Reattachment

After reconnecting, the client can reattach to its previous session. The server loads the session from disk (if evicted from memory) and sends a SessionInfoEvent with the full session state.

What's Preserved

  • Session ID
  • Conversation history (persisted on server disk)
  • Tool states (managed by server)

What's Lost

  • Active IPC connection (replaced by new one)
  • In-flight requests (pending permission responses)
  • Real-time event stream (restarted after reconnect)
Always Call set_session_id()
Always call set_session_id() after creating a session — without it, the recovery client can't reattach after reconnection.
Session tracking
from jaato_sdk.events import ClientType

client = IPCRecoveryClient(
    "/tmp/jaato.sock",
    client_type=ClientType.TERMINAL,
)
await client.connect()

# Create session and track it
# (optionally with an agent profile)
session_id = await client.create_session(
    "work", profile="researcher-claude"
)
client.set_session_id(session_id)

# After reconnection, the client automatically
# sends session.attach with this session_id.
# The server restores conversation history.
Manual reattachment
# If you have a session ID from a previous run
# (e.g., stored in a config file), you can
# reattach manually:

client = IPCRecoveryClient(
    "/tmp/jaato.sock",
    client_type=ClientType.TERMINAL,
)
await client.connect()

# Attach to existing session
success = await client.attach_session(
    "previous-session-id"
)
if success:
    # Track it for future reconnections
    client.set_session_id("previous-session-id")

Configuration

Recovery behavior is configured via RecoveryConfig. Configuration is loaded and merged in precedence order:

  1. Built-in defaults (lowest)
  2. User config (~/.jaato/client.json)
  3. Project config (.jaato/client.json)
  4. Environment variables (highest)

RecoveryConfig Fields

Field Default Description
enabled true Enable automatic reconnection
max_attempts 10 Max reconnection attempts
base_delay 1.0 Initial backoff delay (seconds)
max_delay 60.0 Maximum backoff delay cap
jitter_factor 0.3 Random jitter range (0.3 = ±30%)
connection_timeout 5.0 Timeout per connection attempt
reattach_session true Auto-reattach to previous session

Environment Variables

Variable Config Field
JAATO_IPC_AUTO_RECONNECT enabled
JAATO_IPC_RETRY_MAX_ATTEMPTS max_attempts
JAATO_IPC_RETRY_BASE_DELAY base_delay
JAATO_IPC_RETRY_MAX_DELAY max_delay
JAATO_IPC_RETRY_JITTER jitter_factor
JAATO_IPC_CONNECTION_TIMEOUT connection_timeout
JAATO_IPC_REATTACH_SESSION reattach_session
.jaato/client.json
{
  "recovery": {
    "enabled": true,
    "max_attempts": 10,
    "base_delay": 1.0,
    "max_delay": 60.0,
    "jitter_factor": 0.3,
    "connection_timeout": 5.0,
    "reattach_session": true
  }
}
Config in code
from jaato_sdk.client import (
    IPCRecoveryClient,
    RecoveryConfig,
)
from jaato_sdk.events import ClientType

# Custom config
config = RecoveryConfig(
    max_attempts=5,
    base_delay=2.0,
    max_delay=30.0,
)

client = IPCRecoveryClient(
    socket_path="/tmp/jaato.sock",
    client_type=ClientType.TERMINAL,   # required
    config=config,
)

# Or load from config files automatically
from jaato_sdk.client import get_recovery_config

config = get_recovery_config(
    workspace_path="/path/to/project"
)
client = IPCRecoveryClient(
    socket_path="/tmp/jaato.sock",
    client_type=ClientType.TERMINAL,   # required
    config=config,
)

Python WebSocket Recovery (WSRecoveryClient)

For a remote daemon over WebSocket, the Python SDK ships WSRecoveryClient — a first-class, IPCRecoveryClient-equivalent implementation. It is a subclass of WSClient with the same recovery machinery as the IPC client: the same six-state connection state machine, exponential backoff with jitter, automatic session reattachment, and the on_status_change callback. You get IPC-grade resilience over ws:// / wss:// without dropping to raw frames.

Through the facade, the same .session(...) + ask/complete/stream code runs over WS recovery — either via WSRecoveryClient.session(...) or the transport-agnostic jaato.session(mode="ws", recovery=True, ...) flag. The recovery=True flag works on both daemon transports (mode="ipc" and mode="ws"); it raises ValueError for mode="in_process", which has no daemon to reconnect to.

TLS for self-signed wss://

For a development or internal wss:// endpoint with a self-signed certificate, pass ssl= (an ssl.SSLContext, or True/False) or ca= (a CA-bundle path) — accepted on WSClient / WSRecoveryClient and on jaato.session(mode="ws", ca=...). A ca path is loaded into a default verifying context; ssl wins if both are set. These are scoped per connection — loaded into the connection's SSLContext, never into os.environ — so, unlike an SSL_CERT_FILE env hack, they cannot leak into a subprocess-restarted daemon's outbound HTTPS (the Python analog of Node's NODE_EXTRA_CA_CERTS).

Needs the jaato-sdk[ws] extra
The WS transport lazily imports the websockets package. WSRecoveryClient imports without it, but connecting raises ImportError with an install hint. Install with pip install 'jaato-sdk[ws]'.
Minimal WS recovery client (facade)
import asyncio
import jaato

async def main():
    # Same facade as IPCRecoveryClient, over WebSocket. Reconnection,
    # session creation, and reattachment are handled for you; ask()
    # survives a daemon restart or a dropped socket.
    async with jaato.session(
        mode="ws",
        url="wss://host:8080",
        token="...",                 # bearer token (or ?token= in the url)
        recovery=True,               # → WSRecoveryClient
        ca="/etc/jaato/dev-ca.pem",  # trust a self-signed wss:// cert
        on_status_change=lambda st: print(st.state),  # RECONNECTING / CONNECTED
    ) as s:
        print(await s.ask("Long task..."))

asyncio.run(main())
Constructing the client directly
from jaato_sdk import WSRecoveryClient
from jaato_sdk.events import ClientType

client = WSRecoveryClient(
    url="wss://host:8080",
    token="...",
    client_type=ClientType.API,
    ca="/etc/jaato/dev-ca.pem",   # or ssl=<ssl.SSLContext>
    on_status_change=lambda st: print(st.state),
)
# Same connection-lifecycle API as IPCRecoveryClient:
# connect() / create_session() / send_message() / events().

Browser & Raw-Frame Clients (WebSocket)

Python clients should reach for WSRecoveryClient above. This section documents the underlying raw WS frame protocol for browsers / JavaScript and other non-Python clients (the TypeScript @jaato/sdk recovery: {} wraps these frames for you). The same recovery concepts apply, but reattachment has one extra step that the IPC transport does not. Each WebSocket connection is given its own workspace, so a new connection cannot see a session created on a previous connection until it selects that session's workspace first. The reconnect sequence is session.listworkspace.selectsession.attach.

Key Patterns

  • Use a reconnecting WebSocket library (or your own backoff loop)
  • Wait for the unprompted {"type":"connected"} greeting before sending any command
  • Track the session ID and its workspace_path (read from session.list) client-side
  • Reattach with session.listworkspace.selectsession.attachsession.attach is a command.execute with the session ID as the first positional argument
  • The cold restore is asynchronous: a send issued while it settles can return a recoverable SessionError ("Session not found") — reattach and resend until output arrives
  • Handle the ping/pong keepalive mechanism
WebSocket Authentication Required
The server WS endpoint requires a bearer token on every connection (including reconnects). Python/server-side clients send it as an HTTP header on the Upgrade request: Authorization: Bearer <token>. Browsers cannot set custom headers in new WebSocket(), so pass it as a query parameter instead: ?token=<token>. Omitting or sending an invalid token causes the server to close the connection immediately with WS close code 1008 (Policy Violation). The default token path is $HOME/.jaato/ws.token; it is auto-generated on first server start if not provided via --ws-token / --ws-token-file.
Why the opaque "Session not found"
Attaching a session you have not selected the workspace for returns a bare "Session not found" rather than naming where the session lives. This is deliberate: workspace isolation is a security boundary, and the server will not confirm that a session exists in a workspace the connection has not selected — otherwise the error could be used to enumerate sessions across workspaces. workspace.select is the legitimate way to re-target a workspace you own.
WebSocket reconnection (TypeScript)
// Raw WebSocket reconnect (no SDK). With @jaato/sdk, `recovery: {}` does all of
// this for you; this shows the underlying frames.

class JaatoWebClient {
  private ws: WebSocket | null = null;
  private sessionId: string | null = null;
  private workspacePath: string | null = null; // the session's own workspace
  private reconnectAttempt = 0;
  private maxAttempts = 10;

  // url carries the bearer token: wss://<host>/?token=<token>
  // For a self-signed dev wss://, trust the dev CA out-of-band
  // (e.g. NODE_EXTRA_CA_CERTS), never by disabling verification.
  connect(url: string) {
    this.ws = new WebSocket(url);

    this.ws.onmessage = (e) => {
      const frame = JSON.parse(e.data);
      switch (frame.type) {
        case "connected": // unprompted greeting — only now is it safe to send
          this.reconnectAttempt = 0;
          if (this.sessionId) this.reattach();
          break;
        case "session.list": // find the session's workspace, then select + attach
          this.workspacePath =
            frame.sessions.find((s) => s.id === this.sessionId)?.workspace_path;
          this.send({ type: "workspace.select", name: this.workspacePath });
          this.send({ type: "command.execute",
                      command: "session.attach", args: [this.sessionId] });
          break;
        case "error": // recoverable "Session not found" while the ~15s cold
          if (frame.recoverable) this.reattach(); // restore settles — resend
          break;
      }
    };

    this.ws.onclose = () => this.scheduleReconnect(url);
  }

  // A fresh connection gets its own workspace, so the session created on an
  // earlier connection isn't reachable until we select its workspace. List
  // sessions to find its workspace_path; the handler above does select + attach.
  private reattach() {
    this.send({ type: "command.execute", command: "session.list", args: [] });
  }

  private scheduleReconnect(url: string) {
    if (this.reconnectAttempt >= this.maxAttempts) return; // give up

    const delay = Math.min(60_000, 1000 * Math.pow(2, this.reconnectAttempt));
    this.reconnectAttempt++;
    setTimeout(() => this.connect(url), delay);
  }

  private send(frame: unknown) {
    this.ws?.send(JSON.stringify(frame));
  }
}

Next Steps

Quick reference
# Imports
from jaato_sdk.client import (
    IPCRecoveryClient,
    ConnectionState,
    RecoveryConfig,
    get_recovery_config,
)
from jaato_sdk.client.recovery import (
    ConnectionStatus,
    ReconnectingError,
    ConnectionClosedError,
    IncompatibleServerError,
)
from jaato_sdk.events import ClientType

# States
ConnectionState.DISCONNECTED
ConnectionState.CONNECTING
ConnectionState.CONNECTED
ConnectionState.RECONNECTING
ConnectionState.DISCONNECTING
ConnectionState.CLOSED

# Key methods
client.connect()           # Start connection
client.disconnect()        # Temporary disconnect
client.close()             # Permanent close
client.set_session_id(id)  # Track for recovery
client.send_message(text)  # Send (raises on down)
client.events()            # Auto-reconnecting stream

# Properties
client.state               # ConnectionState
client.is_connected        # bool
client.is_reconnecting     # bool
client.is_closed           # bool
client.session_id          # str | None