mirror of
https://github.com/0xWheatyz/handler.git
synced 2026-08-30 07:16:24 +00:00
Claude
44632ea771
schedules.model_id (migration 0013) pins every fired run of a recurring spawn to a registered model backend, exactly like a hand-spawned agent: the worker copies it into each firing's spawn payload, the launched agent records the pin, and resumes stay on the same backend. The Schedules form gets the same Model dropdown as the spawn form (Claude subscription by default), with a badge in the schedules table. Create/update routes fail fast on a missing or disabled backend so a stale selection bounces immediately instead of every firing failing asynchronously in Activity; a backend deleted later still fails each firing visibly rather than silently falling back to the subscription. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzDofD7gP63WpeLG8vEdZu
106 lines
4.7 KiB
Markdown
106 lines
4.7 KiB
Markdown
# Local model backends (Qwen-Coder & friends)
|
|
|
|
Handler can run agents on locally-hosted models without changing anything about how an
|
|
agent works: it is still the same `claude` binary with the same generated
|
|
`settings.json`, hooks, skills, MCP connectors, plugins, and permission gates. The only
|
|
thing a **model backend** changes is the environment of that one agent's process:
|
|
|
|
| Variable | From |
|
|
|---|---|
|
|
| `ANTHROPIC_BASE_URL` | the backend's `base_url` |
|
|
| `ANTHROPIC_AUTH_TOKEN` | the backend's stored API key (decrypted at launch; a placeholder when none is stored, so the subscription OAuth token is never sent to a local endpoint) |
|
|
| `ANTHROPIC_MODEL` | the backend's `model` |
|
|
| `ANTHROPIC_SMALL_FAST_MODEL` | `small_fast_model`, falling back to `model` |
|
|
| `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` | `1` (skip sidecar calls a local endpoint won't serve; override via the row's env map) |
|
|
|
|
Register backends on the dashboard's **Claude → Models** tab (or `POST /claude/models`),
|
|
then pick one from the **Model** dropdown when spawning an agent — or on a **Schedule**,
|
|
so every fired run spawns on that backend. No selection = the
|
|
worker's logged-in Claude subscription, exactly as before. The agent is *pinned* to its
|
|
backend: resumes come back up on the same one, and deleting a backend makes resumes of
|
|
its agents fail loudly rather than silently falling back to the subscription.
|
|
|
|
## Why "tool calling not working" happens with Qwen-Coder
|
|
|
|
Claude Code speaks the **Anthropic Messages API** (`POST /v1/messages`): it sends tool
|
|
definitions in Anthropic's schema and expects structured `tool_use` content blocks back.
|
|
Local servers — Ollama, llama.cpp's `llama-server`, LM Studio, vLLM's default OpenAI
|
|
mode — speak the **OpenAI Chat Completions API** instead. Point `ANTHROPIC_BASE_URL` at
|
|
one of those and the request either 404s or, with a naive translator in between, the
|
|
model's tool calls come back as *plain text* (Qwen emits its own XML-ish
|
|
`<tool_call>` format) that Claude Code can't execute. That is the whole failure: the
|
|
model is fine, the dialect in the middle is wrong.
|
|
|
|
Two things must both be true:
|
|
|
|
1. **The endpoint must serve the Anthropic Messages API**, translating to whatever your
|
|
server speaks.
|
|
2. **The inference server must parse the model's native tool-call format into
|
|
structured tool calls** — for Qwen that means a Qwen-aware parser/template, not the
|
|
default one.
|
|
|
|
## Working stacks
|
|
|
|
### Recommended: vLLM (Qwen tool parser) + LiteLLM (Anthropic translation)
|
|
|
|
vLLM parses Qwen's tool-call format natively when told to:
|
|
|
|
```bash
|
|
# Qwen3-Coder
|
|
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
|
|
--enable-auto-tool-choice \
|
|
--tool-call-parser qwen3_coder \
|
|
--port 8000
|
|
|
|
# Qwen2.5-Coder uses the hermes parser instead:
|
|
# --tool-call-parser hermes
|
|
```
|
|
|
|
LiteLLM in front exposes the Anthropic `/v1/messages` endpoint:
|
|
|
|
```yaml
|
|
# litellm-config.yaml
|
|
model_list:
|
|
- model_name: qwen3-coder-30b
|
|
litellm_params:
|
|
model: hosted_vllm/Qwen/Qwen3-Coder-30B-A3B-Instruct
|
|
api_base: http://127.0.0.1:8000/v1
|
|
general_settings:
|
|
master_key: sk-local-anything
|
|
```
|
|
|
|
```bash
|
|
litellm --config litellm-config.yaml --port 4000
|
|
```
|
|
|
|
Then register the backend in Handler: base URL `http://<host>:4000`, model
|
|
`qwen3-coder-30b`, API key `sk-local-anything`.
|
|
|
|
### llama.cpp / Ollama
|
|
|
|
- `llama-server` needs `--jinja` (and, for Qwen, a chat template with tool support —
|
|
recent official Qwen GGUFs ship one; older community quants often don't, which is
|
|
another common source of "tools don't work").
|
|
- Ollama supports OpenAI-style tool calling for models whose Modelfile template declares
|
|
it; check `ollama show <model> --template` mentions `.Tools` before blaming the proxy.
|
|
- Either way, they still only speak OpenAI-dialect — keep LiteLLM (use
|
|
`ollama_chat/<model>`, not `ollama/<model>`, for tool support) or
|
|
[claude-code-router](https://github.com/musistudio/claude-code-router) in front as
|
|
the Anthropic translator.
|
|
|
|
## Expectations and tips for small models
|
|
|
|
- **Keep the harness light.** Handler's agents run tool-heavy (hooks, MCP connectors,
|
|
skills). A 7B model will fumble that loop; Qwen3-Coder-30B-class models handle it
|
|
reasonably. Disable connectors the agent doesn't need and keep tasks small and
|
|
concrete.
|
|
- **Raise timeouts, cap output.** The row's env map is the escape hatch:
|
|
`API_TIMEOUT_MS=600000`, `CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192` are sensible for a
|
|
local 30B.
|
|
- **The gates don't relax.** The Stop/PreToolUse hooks still block un-tested,
|
|
un-pushed work regardless of which model produced it — that's the point of keeping
|
|
the same binary.
|
|
- **The subscription is untouched.** The web login, credential sync, and every agent
|
|
spawned without a model selection keep working exactly as before; backends are purely
|
|
additive.
|