Connect a local model

This walkthrough starts a local Ollama server, downloads a small model, and connects it to Haskell Agent. Each stage has a separate verification: server readiness, generated text, model registration, and an actual file-reading tool call.

Verification scope: Ollama 0.32.13 with qwen3:0.6b passed startup, local streaming generation, and a function-call/result replay check on Apple silicon on 22 September 2026. The separate Haskell Agent file-reading exercise below was not executed in that verification. API success is not an end-to-end harness verification.

Prerequisites


- Haskell Agent installed and Nix flakes enabled.
- Two terminals: one remains occupied by the server.
- Internet access for Nix packages and the initial model download.
- About 523 MB for the example model weights, additional space for Nix packages, and sufficient free memory for the model plus a 32,768-token context.

The example uses qwen3:0.6b to keep the initial download small. It is a connection and tool-protocol exercise, not a recommendation for autonomous coding. A small model can answer a greeting while still failing to select or use tools correctly. Use a more capable model after verifying the connection, and review its memory requirements before downloading it.

1. Obtain and start Ollama

In terminal A, enter this Nix environment. The revision pins Ollama to 0.32.13 rather than silently following a changing package registry:

NIXPKGS=github:NixOS/nixpkgs/afe3d8ac4395617bdcdac9f188ac8717a062e014
nix shell "$NIXPKGS#ollama" "$NIXPKGS#curl"

Then start the server in the foreground:

OLLAMA_HOST=127.0.0.1:11434 \
OLLAMA_CONTEXT_LENGTH=32768 \
OLLAMA_NUM_PARALLEL=1 \
OLLAMA_NO_CLOUD=1 \
ollama serve

Leave terminal A open. Expected result: the server reports that it is listening on 127.0.0.1:11434. These settings keep it on loopback, disable Ollama's cloud features, select one concurrent inference request, and explicitly configure the context size used later in the catalog. They do not install a background service.

If the address is already in use, inspect your existing Ollama process instead of starting a second server or terminating an unrelated process. Its existing context and cloud settings may differ. Do not bind an unauthenticated model endpoint to a public network interface.

2. Download and inspect the model

In terminal B, enter the same Nix environment:

NIXPKGS=github:NixOS/nixpkgs/afe3d8ac4395617bdcdac9f188ac8717a062e014
nix shell "$NIXPKGS#ollama" "$NIXPKGS#curl"

Point the client at the server, confirm its version, and download the model:

export OLLAMA_HOST=127.0.0.1:11434
curl --fail --silent --show-error http://127.0.0.1:11434/api/version
ollama pull qwen3:0.6b
ollama list
ollama show qwen3:0.6b

Expected result: ollama pull completes successfully, ollama list includes qwen3:0.6b, and ollama show lists the model's capabilities and context limit. The download is stored by the server, normally under ~/.ollama/models; leaving the Nix shell does not delete it. The upstream model tag can change independently of the pinned Ollama package. Record the model identifier displayed by ollama list when reproducing a problem.

3. Verify the Responses endpoint independently

Run this in terminal B, where curl is already supplied by Nix:

curl --fail-with-body --no-buffer --max-time 120 \
  http://127.0.0.1:11434/v1/responses \
  -H 'Content-Type: application/json' \
  --data '{"model":"qwen3:0.6b","input":"Reply with a short greeting.","stream":true,"store":false,"reasoning":{"effort":"none"}}'

Expected result: an HTTP success response and a stream of Responses events containing generated text, ending in response.completed. The wording may vary. A refused connection means the server is not reachable at that address. A 404 usually means the path or API implementation differs. An unknown-model error means the server's loaded identifier differs from the request.

Ollama's compatibility reference documents Responses support from version 0.13.3. A server offering only /v1/chat/completions is not sufficient. Haskell Agent's portable Responses connection uses stateless streaming; it does not require Ollama to store a conversation through previous_response_id.

For an authenticated endpoint, supply its required authorization header from a secret environment variable. Do not paste the secret into a command saved in shell history. Do not disable TLS verification to make a hosted endpoint work.

4. Register the connection and model

Open ~/.haskell-agent/models.json in your editor. If it already exists, merge the following connection and model into its existing objects and arrays instead of overwriting other entries:

{
  "version": 1,
  "connections": {
    "local-inference": {
      "api": "responses",
      "base_url": "http://127.0.0.1:11434/v1",
      "api_key_optional": true,
      "request_timeout_seconds": 600
    }
  },
  "models": [{
    "id": "local-coder",
    "connection": "local-inference",
    "model": "qwen3:0.6b",
    "dialect": "generic-responses",
    "context_window": 32768,
    "label": "local"
  }]
}

local-coder is the name used by Haskell Agent. The separate model field is sent to the server. generic-responses selects the portable Responses tool interface; it does not convert a Chat Completions server into a Responses server.

The optional-key setting is appropriate only for this explicitly unauthenticated local example. For a protected service, replace it with api_key_env naming the secret variable. See the connection field reference.

5. Test generation through the harness

agent-cli --model local-coder

Run /session-info and verify the selected model, then submit:

Reply with a short greeting. Do not use tools.

Expected result: the selected connection produces a reply. Inspect session information rather than asking the model to identify itself. If direct curl worked but the harness fails, compare the registered URL and wire model name with the successful request, then check startup catalog errors and required environment variables.

6. Test a read-only tool

Use a disposable directory with a known README rather than your main repository. In a separate shell, create the fixture and launch the configured model there:

verification_directory=$(mktemp -d -t local-model-verification.XXXXXX)
printf '# Local model verification\n\nThis is a read-only connection test.\n' > "$verification_directory/README.md"
agent-cli --model local-coder --cwd "$verification_directory" \
  --no-skills --no-agents-md

Submit:

Read README.md using the file-reading tool. Quote its first heading and report the file path. Do not modify files or run shell commands.

Expected result: a visible file-reading tool call followed by an answer based on that result quoting Local model verification. Text claiming to have read the file without a tool call does not establish tool compatibility. A generation-only test can pass while function-call serialization or tool result handling still fails.

If the model produces malformed tool calls, verify the inference server's Responses function-call support and the model's tool-use capability before changing permissions. Approving more operations cannot repair an incompatible API.

The fixture deliberately disables filesystem skills and automatic project instructions. Inspect /mcp and disable any unrelated server you do not want available during the test. These launch flags are not a filesystem sandbox. Do not enable --yolo to compensate for a model that misunderstands the task.

Record this harness check separately from the API checks. A successful protocol exchange does not establish reliable coding behavior or prove that the harness executed the requested file-reading tool.

7. Confirm limits and recovery

Keep context_window aligned with the context actually configured on the server, not merely the model architecture's maximum. Portable-model compaction refuses to guess a missing limit. A larger catalog value does not allocate more server memory.

Failure | Next check | 

Connection refused | Server process, listening address, and port; rerun the direct request. | 

404 or unsupported API | Version prefix and streaming Responses support. | 

401 | Endpoint authentication and the variable named by api_key_env. | 

Timeout | Server logs and model loading. Raise the positive request_timeout_seconds only after establishing that the server is making progress. | 

Tool schema or parsing failure | Responses function calls and model compatibility; do not treat this as an approval failure. | 

Unexpected model selection | Use the local catalog id, restart after editing, and inspect /session-info. | 

Out of memory or prompt truncation | Inspect server logs and ollama ps. Lower the server context and catalog value together, or choose a model and machine with sufficient capacity. | 

To stop using this connection, select another model with /model. Remove only the corresponding model entry and unused connection when editing the catalog. Custom connections are manually selected and are not automatic billing-fallback targets.

8. Stop the server and remove optional data

Exit the test agent with /quit. While Ollama is still running, unload the model; remove its downloaded weights only if you no longer need them:

ollama stop qwen3:0.6b
# Optional: delete this model from the server's model storage.
ollama rm qwen3:0.6b

Press Ctrl+C in terminal A to stop the foreground server. Neither stopping it nor removing weights deletes your Haskell Agent sessions or the catalog entry. Remove that entry separately if it is no longer useful.