Reference
HTTP server
Run the REST and event-stream interface, manage sessions and approvals, and configure secure deployment.
View as textOn this page
- Start a local server
- Create a session and a turn
- Session and turn request fields
- Response schemas and client decoding
- Route reference
- History and session changes
- Approvals and other input
- Events and reconnecting
- SSE envelope and payload reference
- Agent snapshot details
- Nested history records
- History item field reference
- Message content field reference
- Example: display assistant text without replaying tools
- Authentication and network exposure
- All server options
- Multi-tenant deployment
- Provision a tenant safely
- Tenant backups and sandbox recovery
- Embed the Haskell client
- Rotate a server token
- Failure recovery
agent-server exposes the runtime over HTTP. It is an API server, not a browser conversation application. Sessions are durable PostgreSQL records; turn execution and event replay have bounded process-local state.
Start a local server
From a checkout with model credentials configured for the current user:
nix run .#agent-server -- --workspace-root /absolute/path/to/project
# In another terminal:
curl --fail http://127.0.0.1:4096/healthz
curl --fail http://127.0.0.1:4096/readyz
curl --fail http://127.0.0.1:4096/openapi.jsonHealth reports that the HTTP application responds; readiness also checks its backend. A ready server does not establish that every model account has available quota. Without an explicit workspace root, the current directory is the local allowed root. Requested directories are canonicalized; symlinks cannot escape an allowed root.
Create a session and a turn
List models first and substitute an available identifier. Replace the path and SESSION_ID with your project and the returned session identifier:
curl --fail http://127.0.0.1:4096/v1/models
curl --fail -X POST http://127.0.0.1:4096/v1/sessions \
-H 'Content-Type: application/json' \
-d '{"cwd":"/absolute/path/to/project","model":"gpt-5.6-sol"}'
curl --fail -X POST http://127.0.0.1:4096/v1/sessions/SESSION_ID/turns \
-H 'Content-Type: application/json' \
-d '{"input":"Explain the repository. Do not modify files."}'Creation with a repository may return before checkout preparation finishes. Observe session.setup.started, step, completed, or failed events. The first turn waits for setup. Only one turn may be active in a session; independent sessions can run concurrently. Turns move from queued to running, optionally waiting for input, and finally completed, failed, or cancelled.
Session and turn request fields
A basic session request accepts cwd, model, title and effort (none, low, medium, high, xhigh or max, subject to model support). Patch accepts exactly one of title or archived. Fork accepts throughTurn (nonnegative integer), title and cwd; inspect the installed schema and server validation for allowed combinations.
Turn creation accepts input, optional UUID clientRequestId, images and files. Supply a stable client request UUID for idempotent creation and retain it when retrying that same admission. Do not assign a new identifier merely because a network response was lost.
An image entry has mimeType and base64 data; supported MIME types are JPEG, PNG, GIF, WebP and BMP, with at most one image. File entries have name, mimeType and base64 data. Images and files together are limited to five attachments and 20 MiB decoded; encoded JSON must also fit the server's request limit. Opaque files are materialized temporarily under the session directory. Unknown request properties are rejected.
Response schemas and client decoding
Download the server's /openapi.json from the same running release. It describes 17 paths (some support several methods) and 17 named schemas. A local OpenAPI reference copy is bundled with this documentation for offline use; the running server's copy takes precedence when its release differs. Resolve $ref under components.schemas; a nullable field is different from an omitted optional property. Preserve unknown fields when storing objects and tolerate additions in responses.
| Schema | Fields and interpretation |
|---|---|
| Model | Required string id, provider, connection, transportModel, dialect; nullable label and integer contextWindow. Submit the catalog ID, not a guessed transport model. |
| Session | String id, timestamps createdAt/updatedAt, provider/connection/model/dialect/cwd/effort/title; Boolean titleIsManual/archived; nullable transportModel. usage has integer input/output/cached token counts. |
| SessionPage | data array of Session and nullable string nextCursor. A null cursor ends pagination; do not fabricate it from a session identifier. |
| HistoryPage | session, data array of integer index plus turn, integer generationStart/total, Boolean hasOlder/hasNewer, nullable integer nextCursor. Turn items is canonical history; displayItems is rendering-only failed partial output. |
| Turn | UUID id/clientRequestId, string sessionId, status queued/running/waiting_for_input/completed/failed/cancelled, createdAt, nullable startedAt/finishedAt/error, input and userText. Terminal status does not imply success. |
| TurnResult / TurnOutput / TurnCompletion | Result contains turn and nullable output. Output has responseId, nullable assistantText, assistantTextTruncated, completion. Completion status is completed/incomplete, with optional reason and nullable reasoningTokens. Do not present truncated text as the complete canonical transcript. |
| Agent | String path/status, nullable model, array of step objects. Steps are not fully constrained by OpenAPI; render defensively rather than assuming one provider's shape. |
| HumanRequest / ResolveRequest | UUID id/turnId, sessionId, kind, prompt, string options, createdAt. Resolve with required string decision and optional string value. Use the currently advertised request and choices; never infer permission from free-form assistant output. |
| ErrorEnvelope | error contains string code/message/requestId and optional arbitrary details. Retain requestId for diagnostics; redact credentials and private content before sharing details. |
The remaining named request schemas are CreateSession, PatchSession, ForkSession and CreateTurn, described above and in the mutation section. OpenAPI currently leaves nested history turns and agent steps partly open; it is not an exhaustive schema for every provider event. Consult Agent.Server.Types in the matching checkout when implementing those nested objects, and keep SSE decoding separate from REST response decoding.
curl --fail http://127.0.0.1:4096/openapi.json -o agent-server-openapi.json
jq '.components.schemas.TurnResult' agent-server-openapi.json
jq '.paths["/v1/sessions/{sessionId}/turns"]' agent-server-openapi.jsonRoute reference
All paths below are relative to the server URL. The server's /openapi.json is the machine-readable request and response reference for the installed revision. Do not forward arbitrary CLI arguments through JSON.
| Method and path | Operation |
|---|---|
GET /healthz | HTTP health. |
GET /readyz | Backend readiness. |
GET /openapi.json | Installed OpenAPI schema. |
GET /v1/models | Models visible to the authenticated boundary. |
GET /v1/sessions | List sessions using keyset pagination. |
POST /v1/sessions | Create a durable session and optional repository setup. |
GET /v1/sessions/:id | Read a session. |
PATCH /v1/sessions/:id | Rename or archive; one field per request. |
DELETE /v1/sessions/:id | Delete an inactive session. |
GET /v1/sessions/:id/history | Read paginated canonical and display history. |
POST /v1/sessions/:id/fork | Fork current transcript or a durable turn boundary. |
POST /v1/sessions/:id/turns | Queue a typed turn request. |
GET /v1/turns | List retained turn execution records. |
GET /v1/turns/:id | Inspect execution status. |
GET /v1/turns/:id/result | Read the turn result. |
POST /v1/turns/:id/cancel | Cancel queued, running or waiting work. |
GET /v1/turns/:id/agents | Inspect child agents. |
GET /v1/requests | List human-input requests. |
POST /v1/requests/:id/resolve | Resolve an advertised request option. |
GET /v1/events | Subscribe to replayable Server-Sent Events. |
History and session changes
Session listing accepts archive=active and limit=50. History also accepts a limit; use each response's nextCursor as the next request's cursor. History separates canonical items from display-only displayItems. Never send failed display-only output back as model context. Patch one field per request to rename or archive. Mutation and fork requests can return 409 session_busy while a turn is active. Historical forks inherit their title and directory and can be renamed afterward.
Approvals and other input
Fetch /v1/requests, inspect the complete prompt and advertised options, and resolve the intended request. Use the option supplied by that request rather than assuming every request accepts the same decision. For an approval advertising allow_once:
curl --fail http://127.0.0.1:4096/v1/requests
curl --fail -X POST http://127.0.0.1:4096/v1/requests/REQUEST_ID/resolve \
-H 'Content-Type: application/json' -d '{"decision":"allow_once"}'Plan feedback can include a value. Requests over 64 KiB encoded JSON or 100 options are rejected, not truncated. Do not approve an unseen suffix. If a request disappeared or was resolved elsewhere, refresh it rather than replaying a stale decision. Cancel work with a POST to its turn's /cancel; cancellation does not undo completed file or remote-service mutations.
Events and reconnecting
curl --no-buffer http://127.0.0.1:4096/v1/events
# Reconnect using the last SSE id that your client applied:
curl --no-buffer -H 'Last-Event-ID: 42' http://127.0.0.1:4096/v1/eventsPersist the last applied event identifier. On replay.reset, refetch sessions, turns and requests before continuing: retention expiry or a slow consumer can invalidate incremental state. Retry/discard/failure/tool-retraction events include display-only boundaries. The latest 1,000 terminal turn records are retained in process memory; durable history is separate. Do not depend on a server restart preserving an active turn or the same replay window.
SSE envelope and payload reference
Ordinary SSE records contain an integer id, an event name and JSON data. That JSON is the envelope: id, type, nullable string turnId and sessionId, timestamp at, and the nested data payload described below. Unknown future event names should not crash the client. Apply known events before advancing your saved cursor; do not mistake a keepalive comment for an application event.
replay.reset is a control exception: its data is directly {"reason":"event_gap","refetch":true}, with no ordinary envelope or new cursor. Discard incremental assumptions and refetch before applying subsequent events. Checkout step statuses are running, completed and failed.
| Event names | Nested data fields |
|---|---|
turn.queued, turn.started, agent.turn.started | Empty object. Supervisor turn lifecycle differs from individual agent/model turn lifecycle. |
turn.completed, turn.failed, turn.cancelled | Nullable string error; fetch the turn result rather than treating this as its full output. |
response.text.delta, response.reasoning.delta, turn.activity, warning | String text, Boolean truncated. Plan deltas also use response.text.delta. |
provider.limit | String text, Boolean warning and truncated. |
response.restarted | String reason, Boolean truncated, displayOnly: true. |
agent.turn.finished | String responseId, nullable string assistantText, Boolean assistantTextTruncated, usage, completion. |
tool.started, tool.updated, tool.arguments.updated | Strings callId, name, kind; Booleans argumentsEncrypted, async, argumentsTruncated; nullable string arguments, null when encrypted. |
tool.output.updated | Strings name, output, Boolean truncated. |
tool.finished | Strings callId, kind, output; Booleans async, truncated; integer imageCount. Image URLs/data are deliberately omitted. |
tool.retracted | String callId and displayOnly: true. |
response.attempt.discarded, response.attempt.failed, model.context.reset | displayOnly: true; update presentation state, never append discarded material to canonical model history. |
agent.started | Strings agentId, label; nullable strings parentId, model. |
agent.output | Strings agentId, output, Boolean truncated. |
agent.finished | String agentId; status is running, completed, failed or cancelled. |
request.created | The HumanRequest object described above. |
request.resolved | String requestId; remove that pending request and refetch if your state is stale. |
session.setup.started, session.setup.completed | Strings repository and branch. |
session.setup.step | Checkout operation id, command and status. |
session.setup.failed | String message; inspect checkout diagnostics before another setup attempt. |
usage contains integer inputTokens, outputTokens and cachedTokens. Completion has status: completed, or status: incomplete with string reason and nullable reasoningTokens. Tool kinds are function, custom, computer or computer_function. Public event text is bounded to 16,383 characters before an ellipsis; honor each truncation flag rather than presenting a shortened result as complete.
Agent snapshot details
Each public agent has string path and status, nullable string model, and steps. Each step has state (running, completed, failed or info), string title and nullable string detail. At most 100 agents and 100 steps per agent are projected. Transcripts and retained UI state are deliberately omitted, not missing data that another query parameter can enable.
The public JSON projection has a 64 Ki-character text budget and 2,048-node budget. It redacts encrypted-content/encrypted-function-argument keys and may mark an object projectionTruncated: true. Clients must tolerate shortened arrays and omitted fields; this endpoint is a bounded status view, not a lossless session export.
Nested history records
Each history data entry has an integer index and a turn object. The turn contains at (UTC timestamp), userText, nullable assistantText, error, responseId, effect, items, displayItems, nullable usage, and providerTelemetry. Effects are append, replace or reset; do not flatten them into an append-only model transcript. Usage here uses input, output, cached, unlike SSE's token-counter names.
On the latest page, queued/running/waiting turns with nonblank input can appear as provisional entries with an extra status field, empty item arrays, null usage and no assistant text. Their indices and the overlaid total are not a substitute for durable completion. Refetch after completion rather than appending a second copy of the provisional prompt.
Both item arrays contain tagged Responses objects, not plain strings. Messages use type: message, role and content; function calls use call_id, name and a JSON-encoded arguments string; outputs correlate by call_id. Other tags represent custom/computer calls, reasoning, references, agent messages, additional tools, local shell, tool search, web search, image generation and compaction. Preserve unknown tags as opaque data rather than treating them as assistant text or executable instructions. Content parts can represent images/files as well as text; do not automatically fetch embedded URLs.
This is a bounded public projection, not a lossless provider archive: encrypted-content/encrypted-function-argument keys are redacted, strings and nested structures can be cut, and an object can carry projectionTruncated: true. The recursive budget is 65,536 text characters and 2,048 nodes; individual strings are limited to 16,383 characters. Clients must tolerate missing/truncated nested fields and never replay this projection as canonical model input. Redaction matches encrypted_content, encryptedcontent, encrypted_function_args and encryptedfunctionargs, case-insensitively, replacing their values with <redacted>. Ordinary arguments, text and tool output are not secret-scrubbed. Apply the session's access controls to the whole response.
History item field reference
These are the current typed encodings in Agent.Responses.Types.Items, not a closed provider schema. A question mark below means optional before projection; after projection, even normally present fields can be absent. Unless stated otherwise, identifiers, names, status values and textual payloads are strings. Item status usually means in_progress, completed or incomplete; preserve unfamiliar values instead of mapping them to success. Item completion is not the same as completion of the enclosing turn.
| Item type | Fields before public projection |
|---|---|
message | id?, role, content (string or parts), status?, phase?, internal_chat_message_metadata_passthrough?. |
agent_message | id?, author?, recipient?, content (parts), internal_chat_message_metadata_passthrough?. |
function_call | id?, call_id, name, namespace?, provider?, arguments (JSON-encoded string), encrypted_function_args? (string array before redaction), status?, async? (boolean). |
function_call_output | id?, call_id, name?, namespace?, provider?, output (arbitrary JSON), status?, async? (boolean). Local execution outcome is not serialized here. |
custom_tool_call | id?, call_id, name, namespace?, input (string), status?, async? (boolean). |
custom_tool_call_output | id?, call_id, name?, output (arbitrary JSON), status?, async? (boolean). |
computer_call | id?, call_id, actions (array), pending_safety_checks? (array), status?; extension fields may also be retained. |
computer_call_output | id?, call_id, output ({type: computer_screenshot, image_url: string, detail: original}), acknowledged_safety_checks? (array), status?; extension fields may also be retained. |
reasoning | id?, summary (array of {type: string, text?: string}), content? (parts), encrypted_content? (redacted), status?. |
item_reference | id. A reference does not contain the referenced item's body. |
additional_tools | id?, role, tools (array of arbitrary JSON tool definitions). |
local_shell_call | id?, call_id?, status?, action?. The exec action has command (string array), timeout_ms? (integer), working_directory?, env? (string-to-string object), user?. |
tool_search_call | id?, call_id?, status?, execution?, arguments? (arbitrary JSON, unlike function_call.arguments). |
tool_search_output | id?, call_id?, status?, execution?, tools (array of arbitrary JSON). |
web_search_call | id?, status?, action?. search has query? and queries? (string array); open_page has url?; find_in_page has url? and pattern?. |
image_generation_call | id?, status?, revised_prompt?, result? (string). Treat the result as provider data, not an automatically displayable image. |
compaction / context_compaction | id?, encrypted_content? (redacted). The decoder accepts compaction_summary as an alias of compaction; encoding normalizes it to compaction. |
compaction_trigger | No additional typed fields. |
Computer actions are tagged objects: screenshot and wait have no additional typed fields; click has integer x, y, string button and string-array keys; double_click and move have coordinates and keys; type has text; keypress has keys; scroll adds integer scroll_x and scroll_y; drag has path, an array of integer x/y points, and keys. A safety check has id, optional code and message, with extension fields permitted. Historical actions and acknowledgments are display records, never permission to perform them again.
Message metadata internal_chat_message_metadata_passthrough, when present, contains optional turn_id, arbitrary JSON create_time and executed_tool_calls, and string-array content_item_kinds. Do not use these internal hints as an authorization or stable pagination contract.
The registry also recognizes file_search_call, code_interpreter_call, local_shell_call_output, shell_call, shell_call_output, apply_patch_call, apply_patch_call_output, mcp_list_tools, mcp_approval_request, mcp_approval_response, mcp_call, program and program_output. They have no typed payload model here: the current TaggedObject decoder/encoder retains only type. Unknown item, content and action tags use the same type-only fallback. Do not invent fields from another provider's API manual or claim that decoding and re-encoding retains arbitrary provider data. A client may retain the raw JSON it actually receives, subject to its own storage policy, but cannot recover fields already discarded upstream.
Message content field reference
A message's content is either a string or an array of tagged parts. Agent-message content is an array. Reasoning content, when present, is also an array. Roles include user, assistant, system and developer, with unknown roles retained. Keep these roles distinct in the display; a historical system message does not instruct the client application.
| Content type | Fields before public projection |
|---|---|
input_text | text; prompt_cache_breakpoint? (arbitrary JSON). |
output_text | text; annotations? and logprobs? (arrays of arbitrary JSON). |
text / reasoning_text / summary_text | text. Preserve the content type when choosing a display region. |
refusal | refusal (string); do not infer refusal from text heuristics. |
input_image | detail?, file_id?, image_url?, prompt_cache_breakpoint? (arbitrary JSON). |
input_file | detail?, file_data?, file_id?, file_url?, filename?, prompt_cache_breakpoint? (arbitrary JSON). |
input_audio | input_audio (arbitrary JSON). |
encrypted_content | encrypted_content (redacted); no displayable plaintext is supplied. |
Raw JSON fields such as annotations, log probabilities, audio descriptors, search tools and tool outputs intentionally have no exhaustive nested schema in this library. Display only shapes your application implements; offer an unsupported content placeholder otherwise. Never automatically navigate a URL, render supplied HTML, decode an unbounded image, play audio or execute a tool from history.
Example: display assistant text without replaying tools
Save the following as HistoryDisplay.hs and load it with nix develop .#docs -c ghci HistoryDisplay.hs. Pass a decoded item from the history response to assistantText; render each returned Text through your UI's escaping API, not as HTML. Empty output means “no supported assistant text,” not an empty successful response.
{-# LANGUAGE OverloadedStrings #-}
module HistoryDisplay (assistantText) where
import Data.Aeson (Value(..))
import qualified Data.Aeson.KeyMap as KeyMap
import Data.Foldable (toList)
import Data.Text (Text)
assistantText :: Value -> [Text]
assistantText (Object item)
| KeyMap.lookup "type" item == Just (String "message")
, KeyMap.lookup "role" item == Just (String "assistant") =
case KeyMap.lookup "content" item of
Just (String text) -> [text]
Just (Array parts) -> concatMap displayPart (toList parts)
_ -> []
assistantText _ = []
displayPart :: Value -> [Text]
displayPart (Object part)
| Just (String kind) <- KeyMap.lookup "type" part
, kind `elem` ["output_text", "text"]
, Just (String text) <- KeyMap.lookup "text" part = [text]
displayPart _ = []
- Fetch a history page and retain its turn indices. Refetch provisional entries after completion; do not blindly append them again.
- Check truncation markers and show an incomplete-record notice. Keep
itemsanddisplayItemsseparate rather than assuming both arrays form one canonical transcript. - For a normal assistant message, the example returns its string content or supported text parts. It ignores calls, tool output, images, reasoning and unknown tags; give those separate, non-executing UI representations where implemented.
- Test at least a text message, unknown tag, non-object item, missing content, image part, redacted value and truncated object. None should trigger a network fetch, tool execution or failure of the entire page.
This is a source-reviewed display example, not an authenticated server-client integration test. Pin the runtime revision and revisit the tables when its Items, Items.Known or Content types change.
Telemetry entries contain nullable duration_ms, api_duration_ms, cost_usd, stop_reason, provider_turns, structured_output, and a models map. Each model entry has input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens, and nullable web_search_requests, cost_usd, context_window, max_output_tokens, canonical_model, provider. Null means unreported, not zero cost.
Authentication and network exposure
Default loopback mode validates the Host header and rejects browser origins unless explicitly allowed. A non-loopback bind requires --allow-remote and authentication. In single-user mode choose exactly one token source: AGENT_SERVER_TOKEN or --token-file. A token on loopback also enables bearer authentication. Use a regular non-symlink owner-only token file; do not put a secret literal in shell history, a URL, or the Nix store.
nix run .#agent-server -- --host 127.0.0.1 --token-file "$HOME/.config/haskell-agent/server-token"Clients send Authorization: Bearer …. The server does not terminate TLS. Keep it behind trusted TLS termination, preserve authentication and SSE streaming, and configure the exact permitted browser origin. CORS is not authentication. Organization gateway identity is an additional boundary; switching credentials invalidates operations that no longer belong to the admitted identity.
All server options
Flags are read at startup. Numeric capacities must be positive; tenant limits cannot exceed their corresponding global limits. Restart with a reviewed configuration to change them. Paths below must satisfy the relevant ownership and canonical-path checks.
| Option | Default | Meaning |
|---|---|---|
--host | 127.0.0.1 | Listening address. |
--port | 4096 | Port 1–65535. |
--allow-remote | false | Permit non-loopback with authentication. |
--token-file | none | Owner-only single-user bearer file; alternative to AGENT_SERVER_TOKEN. |
--tenant-registry | none | Enable versioned multi-tenant credential registry. |
--tenant-state-root | none | Server-owned tenant storage. |
--sandbox-runner | none | Trusted runner; use the NixOS service boundary. |
--yolo | false | Auto-approve server-turn mutations; distinct from sandbox execution policy. |
--cors-origin | none | Repeat for each explicitly allowed browser origin. |
--workspace-root | current directory | Repeatable canonical local workspace root; registry controls tenant workspaces. |
--max-concurrent-turns | 3 | Global running-turn capacity. |
--max-concurrent-turns-per-tenant | 2 | Per-tenant running capacity. |
--max-queued-turns | 100 | Global queue capacity. |
--max-queued-turns-per-tenant | 25 | Per-tenant queue capacity. |
--max-active-tenants | 16 | Active tenant runtime capacity. |
--max-event-subscribers | 256 | Global SSE connections. |
--max-event-subscribers-per-tenant | 8 | Per-tenant SSE connections. |
--event-replay-limit | 1000 | Events retained per access boundary. |
--maximum-request-bytes | 33554432 | Maximum encoded JSON request body (32 MiB). |
Multi-tenant deployment
Use the exported nixosModules.agent-server module, not a hand-launched sandbox runner. It establishes the dedicated account, trusted immutable runner, cgroup delegation and service confinement that the runner checks. Configure services.haskell-agent.server with enable, tenantRegistryFile, explicit workspaceRoots, and sufficient maxActiveTenants. Provision credentials outside the Nix store.
{"version":1,"tenants":[{"id":"018f6a14-7d52-7a52-9c00-66d5e7d70334","workspaceRoot":"/srv/agent-workspaces/acme","credentials":[{"id":"018f6a14-7d52-7a52-9c00-66d5e7d70335","tokenFile":"/run/credentials/acme-agent-token"}]}]}Tenant and credential IDs are canonical UUIDs. Registry and credential files are regular non-symlink mode-0600 files owned by the service user. Tokens contain at least 32 bytes and are unique. Workspace roots must exist, be non-overlapping, match the module allowlist, and exclude credentials and server state. Parent paths must be root- or server-owned and not group/other writable. Use canonical absolute non-root paths without dot components or systemd percent specifiers.
Each tenant receives a separate database and restricted database role. Model-controlled execution runs in a tenant gVisor sandbox with writable workspace and guest state; provider and database credentials remain on the host. Sandbox tools are auto-approved by default, but host mutations such as MCP retain their approval policy. Plan and dangerous-command restrictions still apply. Network access means a shell command can have external effects even inside a sandbox: do not provision unintended production credentials there. Failed sandbox admission never falls back to host execution.
Operators still own TLS, rate limits, disk/database quotas and backups. Each sandbox process tree is limited to two CPUs, 2 GiB RAM without swap and 512 processes. Outbound networking denies private/host/metadata destinations; no inbound service or SSH is provided. Review the repository's docs/agent-server.md and nix/modules/agent-server.nix when deploying the pinned revision.
Provision a tenant safely
- Pin a supported Linux flake revision and import
nixosModules.agent-server. Set the service's registry path and explicit workspace allowlist; retain the dedicated defaulthaskell-agent-serveruser/group unless you deliberately provision equivalent isolation. - Create disjoint canonical workspace directories and private runtime secret files before starting the service. Generate independent high-entropy tokens through your secret manager, not literal Nix strings. Install registry and token files as service-owned mode 0600 regular files, with protected parents. Use the registry example above with newly assigned canonical tenant and credential UUIDs.
- Deploy the reviewed NixOS configuration. The module supplies the private state directory (default
/var/lib/haskell-agent-server), trusted generation-specific runner and cgroup confinement. Do not copy a mutable runner into place or launch it manually to bypass admission. - Before enabling external traffic, check readiness and authenticated protected reads for each tenant using privately provisioned clients. Confirm an invalid token is rejected and each tenant sees only its own sessions. Exercise sandbox admission with a bounded read-only task; this is a required deployment check, not a test claimed here.
- Configure TLS, rate limits, quotas and backups, then allow clients to submit work. Registry/token changes should use a controlled stop, secret deployment and restart; do not assume hot reload.
Tenant backups and sandbox recovery
Back up the tenant workspace, persistent tenant state (including its home/guest state), and PostgreSQL database with its ownership/grants as a consistent recovery set. Preserve tenant UUIDs, registry mapping and the pinned service revision separately; protect credentials in the secret manager. A workspace-only backup cannot restore session history, and an SSE cursor cannot restore a process-memory turn.
- Quiesce submissions, settle or cancel active turns, reconcile possible external effects, and stop the service before filesystem snapshots. Use your PostgreSQL backup tooling for database consistency, not a live copy of database files.
- Restore first in an isolated environment at the matching revision. Restore workspace/state ownership and database roles/grants without broadening cross-tenant access. Re-provision private registry/token files; do not restore obsolete exposed tokens into production.
- Validate readiness, tenant isolation, durable session/history reads and sandbox admission before reopening submissions. Reconnect clients and refetch state; do not blindly replay pre-failure POST requests.
If sandbox startup fails, inspect service logs, exact runner generation, path ownership and the delegated cpu/memory/pids cgroup boundary. The current runner is gVisor, not a microVM. A stale tenant cgroup blocks replacement launches. If cleanup cannot prove descendant quiescence, it fail-stops while retaining the tenant lock until the supervisor kills the complete process group. Stop and investigate the service boundary; never delete a live lock or enable host execution as a workaround. These are operator recovery procedures, not a claim of a tested backup/restore system.
Embed the Haskell client
Add the pinned agent-server-client package to your application's Cabal dependencies. The HTTP client is separate from the server executable. Provision an owner-only credential file for the intended server identity, then construct the client once and reuse it. This source-reviewed example lists pending human requests without approving them:
{-# LANGUAGE OverloadedStrings #-}
import Agent.Server.Client
main :: IO ()
main = do
created <- newAgentServerClient AgentServerClientConfig
{ agentServerBaseUrl = "http://127.0.0.1:4096"
, agentServerCredentialFile = "/absolute/private/server-token"
}
case created of
Left err -> print err
Right client -> listAgentServerRequests client >>= printConstruction validates the URL and reads the credential file; it does not prove remote authorization. Never log the token or redirect an authenticated request to another authority. Handle AgentServerCredentialError, AgentServerTransportError, AgentServerHttpError, AgentServerDecodeError and AgentServerProtocolError separately: fixing local credentials differs from retrying a read after transport failure, and a decoding error can indicate a client/server revision mismatch.
The client exports create-session/create-turn, turn/result/list/cancel, request-list/request-list-for-turn/resolve, history and turn-stream operations. Their request/response records are re-exported from Agent.Server.Client.Protocol. Use streamAgentServerTurn client turnId lastEventId onEvent with a callback that returns an error when it cannot apply an event. Handle completed, failed, cancelled and AgentServerStreamNeedsRefetch distinctly. The refetch value is the last delivered event id; pass it as lastEventId on the next subscription so earlier events, including a consumed approval, are not applied again. Refetch durable history/state after a replay gap instead of presenting a partial transcript as complete. Polling a result is not equivalent to resolving an approval. Resolving an approval that was already consumed returns a conflict, not a missing request.
Rotate a server token
- Schedule an interruption; stop new submissions and let existing turns settle or cancel them and reconcile effects. Do not assume a token reload endpoint.
- Stop the server. Replace its configured token file using a secure secret deployment mechanism, retaining required ownership and mode; if using the environment source, replace that secret instead, not both sources.
- Update authorized clients' secret stores and restart the server. Recreate Haskell client objects because construction reads their bearer token.
- Verify a protected read succeeds with the new credential and rejects the old one. Health alone is not an authentication test. Reconnect event streams and reconcile current state before resuming submissions.
This deliberately bounded maintenance procedure does not promise zero-downtime rotation. Preserve incident evidence if a credential was exposed and inspect previous requests and remote mutations rather than merely replacing the secret.
Failure recovery
- 401: check the chosen token source and authenticated tenant, not model credentials.
- 403 or rejected workspace: check origin/Host and canonical workspace ownership.
- 409 session_busy: inspect the active turn, wait or deliberately cancel it.
- Queue/subscriber limit: reduce concurrency; do not blindly resubmit a mutation.
- Readiness failure: inspect service logs and database access before accepting work.
- Uncertain POST outcome: query the session/turn before repeating an external action.
Errors contain error.code, error.message and error.requestId. Retain the request ID, but redact tokens and sensitive prompt/tool content from reports. These procedures are source-verified, not a claim that a production tenant deployment was exercised during documentation checks.