Merge pull request #20 from 0xWheatyz/feat/headless-runner

.gitignore

@@ -220,3 +220,6 @@ __marimo__/
 # Streamlit
 .streamlit/secrets.toml
 .omc/
+
+# Handler runtime artifacts
+/handler.db

README.md

@@ -13,10 +13,11 @@ git remote, and your own network exposure.
 > control layer, HTTP API, database, migrations, and verification/approval hooks are
 > implemented and tested (106 tests, SQLite). Phase 2 adds credential resolution +
 > injection, role-based forge-workflow skills, a hard approval gate, and a CI-status
-> poller. Live end-to-end agent spawning against a real `claude` binary + tmux is stubbed
+> poller. Agent runs are headless (`claude -p --output-format stream-json`, supervised
-> behind mockable seams (`tmux`, `verify`, `forge`, `gitops`, `spawn.resume`) and wired
+> by the worker, events persisted to the DB); the run/kill/resume paths are exercised
-> but not yet exercised against production binaries. See [`docs/PLAN.md`](docs/PLAN.md)
+> end-to-end against a scripted fake claude binary, with a manual validation script
-> for the full design and roadmap.
+> (`scripts/validate_claude_headless.sh`) for the real one. See
+> [`docs/PLAN.md`](docs/PLAN.md) for the full design and roadmap.
 
 ---
 
@@ -45,19 +46,23 @@ the control layer and API are disposable compute that can restart or scale out f
 ```
         writes                                        reads (+ answer backfill)
   ┌──────────────────┐        ┌──────────────┐        ┌──────────────────┐
-  │  control layer   │───────▶│   database   │◀───────│    HTTP API      │
+  │  worker(s)       │───────▶│   database   │◀───────│    HTTP API      │
   │  (CLI + hooks)   │        │  PG / SQLite │        │  (FastAPI)       │
   └──────────────────┘        └──────────────┘        └──────────────────┘
      │        ▲                                              ▲
      │ spawns │ Stop / PreToolUse / Notification hooks       │ curl, UI, any client
-     ▼        │ write checkmark + log rows                   │ (bearer token)
+     ▼        │ + streamed run events, checkmark, log        │ (bearer token)
-  tmux + claude binary (one working dir / worktree per agent)
+  claude -p --output-format stream-json (one working dir / worktree per agent)
 ```
 
-- **Control layer** (`handler.control`) — the only writer. Spawns/lists/attaches/kills
+- **Control layer / workers** (`handler.control`) — the only writer. Runs each agent as
-  agents as `tmux` sessions running the `claude` binary, one working directory or git
+  a **headless** `claude -p --output-format stream-json` subprocess (one working
-  worktree per agent, namespaced `project__agent`. Stateless; every write goes straight
+  directory or git worktree per agent), streams every stdout event into the database as
-  to the database.
+  it happens, and reconciles agent status from the process itself (exit code + EOF —
+  positive liveness, no screen scraping). Stateless: repo state is pulled from git when
+  a task is claimed, claude session transcripts are archived to / materialized from the
+  DB for cross-worker `--resume`, and the claude login credential bundle is distributed
+  encrypted through the DB. tmux survives only to drive the interactive `/login` flow.
 - **Hooks** (`handler.hooks`) — run inside each agent via a generated `settings.json`.
   They write the checkpoint/log rows and enforce the test and push gates.
 - **API** (`handler.api`) — a thin, read-mostly HTTP layer over the same database (the
@@ -77,10 +82,32 @@ The data model is defined once (SQLAlchemy Core) and renders correctly on both:
 Portable column types bridge the two, and the checkmark upsert uses native
 `INSERT … ON CONFLICT DO UPDATE` on both dialects. Migrations are Alembic, dual-dialect.
 
+### Scaling workers horizontally
+
+Multiple worker containers can drain the same command queue concurrently (Postgres
+`FOR UPDATE SKIP LOCKED`); each supervises up to `MAX_CONCURRENT_RUNS` claude processes
+and skips claiming run-starting commands while full, leaving them for a less-loaded
+worker. Workers heartbeat into the DB; if one dies mid-run, any surviving worker's
+reaper marks its runs (and their agents) `crashed` — visible in the UI with the last
+output preserved — and the operator resumes explicitly on whichever worker picks it up.
+
+Deployment invariants for multi-worker:
+
+- **No shared filesystems.** Git carries repo state (workers clone/pull on claim);
+  claude session transcripts live in `session_archives`; login credentials are
+  Fernet-encrypted into `runtime_secrets` and materialized by every worker.
+- **Identical `PROJECTS_ROOT` on every worker** — claude keys its session storage to
+  the absolute working-dir path, so cross-worker `--resume` needs the same layout.
+- **The same `HANDLER_SECRET_KEY` on every worker** (and the API) — without it, the
+  credential bundle can't be distributed and only the worker that ran `/login` can run
+  agents.
+- The two-step web login is automatically pinned to one worker
+  (`commands.target_worker`), so it works unchanged with a fleet.
+
 ## Requirements
 
 - Python 3.11+
-- `git` and `tmux` (for live spawning)
+- `git` (for live spawning) and `tmux` (only for the web `/login` flow)
 - A `claude` binary, authenticated (for live spawning)
 - `mise` in each managed project, with a `.mise.toml` defining at least a `test` task
 - Postgres (default) — or nothing but a file path for the SQLite fallback

frontend/components/sections/AgentsSection.tsx

@@ -144,11 +144,12 @@ export function AgentsSection() {
                             </div>
                           </td>
                         </tr>
-                        {a.status === "working" && a.last_output?.trim() && (
+                        {(a.status === "working" || a.status === "crashed") && a.last_output?.trim() && (
                           <tr>
                             <td colSpan={6} style={{ paddingTop: 0 }}>
                               <div className="faint" style={{ fontSize: "var(--text-xs)", marginBottom: 4 }}>
-                                live output{a.output_at ? ` · ${timeAgo(a.output_at)}` : ""}
+                                {a.status === "crashed" ? "last output before crash" : "live output"}
+                                {a.output_at ? ` · ${timeAgo(a.output_at)}` : ""}
                               </div>
                               <pre
                                 className="mono"

frontend/components/sections/RunsSection.tsx

@@ -7,13 +7,14 @@ import { useMemo, useState } from "react";
 import { useDashboard } from "@/components/store";
 import { Badge, Button, Callout, Stat, StatusBadge, Tabs, Textarea } from "@/components/ui";
 import { fmtFull, shortSha, statusTone, timeAgo } from "@/lib/format";
-import type { Agent } from "@/lib/api";
+import type { Agent, AgentEvent } from "@/lib/api";
 
 const FILTERS = [
   { value: "all", label: "All" },
   { value: "needs", label: "Needs Input" },
   { value: "working", label: "Working" },
   { value: "done", label: "Done" },
+  { value: "crashed", label: "Crashed" },
 ];
 
 function matches(filter: string, status: string): boolean {
@@ -21,6 +22,7 @@ function matches(filter: string, status: string): boolean {
   if (filter === "needs") return status === "paused_for_input";
   if (filter === "working") return status === "working" || status === "running";
   if (filter === "done") return status === "done" || status === "completed";
+  if (filter === "crashed") return status === "crashed" || status === "blocked";
   return true;
 }
 
@@ -239,6 +241,40 @@ function RunDetail() {
           </div>
         )}
 
+        {/* Headless run event stream (empty for legacy tmux agents) */}
+        {agent?.session_id && (
+          <div style={{ display: "flex", flexDirection: "column", gap: 10 }}>
+            <div className="eyebrow">
+              Run events
+              {agent.worker_id ? (
+                <span className="faint mono" style={{ fontSize: "var(--text-xs)", marginLeft: 8 }}>
+                  on {agent.worker_id}
+                </span>
+              ) : null}
+            </div>
+            {s.events.length === 0 ? (
+              <div className="empty">No events yet.</div>
+            ) : (
+              <div
+                style={{
+                  display: "flex",
+                  flexDirection: "column",
+                  gap: 6,
+                  maxHeight: 420,
+                  overflow: "auto",
+                  padding: "10px 12px",
+                  background: "var(--surface-2, rgba(0,0,0,0.25))",
+                  borderRadius: 6,
+                }}
+              >
+                {s.events.map((e) => (
+                  <EventLine key={e.id} e={e} />
+                ))}
+              </div>
+            )}
+          </div>
+        )}
+
         {/* Log */}
         <div style={{ display: "flex", flexDirection: "column", gap: 10 }}>
           <div className="eyebrow">Log · newest first</div>
@@ -304,3 +340,99 @@ function RunDetail() {
     </>
   );
 }
+
+/* One stream-json event, rendered by type: assistant text as prose, tool calls as chips,
+ * the result as a cost/turns footer, worker notices as callouts, raw lines verbatim. */
+function EventLine({ e }: { e: AgentEvent }) {
+  const p = (e.payload ?? {}) as Record<string, any>;
+  const xs = { fontSize: "var(--text-xs)" } as const;
+
+  if (e.type === "system") {
+    return (
+      <div className="faint mono" style={xs}>
+        ▸ session {p.subtype ?? "event"}
+        {p.session_id ? ` · ${String(p.session_id).slice(0, 8)}` : ""}
+        {Array.isArray(p.tools) ? ` · ${p.tools.length} tools` : ""}
+      </div>
+    );
+  }
+  if (e.type === "assistant") {
+    const content = p.message?.content;
+    const blocks: any[] = Array.isArray(content) ? content : [];
+    const text = blocks
+      .filter((b) => b?.type === "text" && b.text)
+      .map((b) => b.text)
+      .join("\n");
+    const tools = blocks.filter((b) => b?.type === "tool_use");
+    return (
+      <div style={{ display: "flex", flexDirection: "column", gap: 4 }}>
+        {text && (
+          <div style={{ fontSize: "var(--text-sm)", whiteSpace: "pre-wrap" }}>{text}</div>
+        )}
+        {tools.length > 0 && (
+          <div className="hstack" style={{ gap: 6, flexWrap: "wrap" }}>
+            {tools.map((t, i) => (
+              <Badge key={i} tone="info">
+                {t.name}
+                {t.input ? `: ${oneLine(t.input)}` : ""}
+              </Badge>
+            ))}
+          </div>
+        )}
+      </div>
+    );
+  }
+  if (e.type === "result") {
+    const err = Boolean(p.is_error);
+    return (
+      <div className="hstack" style={{ gap: 8, flexWrap: "wrap" }}>
+        <Badge tone={err ? "danger" : "success"}>{err ? "run errored" : "run finished"}</Badge>
+        <span className="faint mono" style={xs}>
+          {p.num_turns != null ? `${p.num_turns} turns` : ""}
+          {p.total_cost_usd != null ? ` · $${Number(p.total_cost_usd).toFixed(4)}` : ""}
+        </span>
+        {typeof p.result === "string" && p.result && (
+          <span className="muted" style={{ ...xs, whiteSpace: "pre-wrap", width: "100%" }}>
+            {p.result}
+          </span>
+        )}
+      </div>
+    );
+  }
+  if (e.type === "worker") {
+    return (
+      <Callout tone="danger">
+        {p.notice ?? "runner notice"}
+        {p.stderr_tail ? (
+          <pre className="mono" style={{ ...xs, margin: "6px 0 0", whiteSpace: "pre-wrap" }}>
+            {p.stderr_tail}
+          </pre>
+        ) : null}
+      </Callout>
+    );
+  }
+  if (e.type === "raw") {
+    return (
+      <div className="faint mono" style={{ ...xs, whiteSpace: "pre-wrap" }}>
+        {typeof p.line === "string" ? p.line.trimEnd() : JSON.stringify(p)}
+      </div>
+    );
+  }
+  // user (tool results) and anything future: a quiet one-liner, nothing lost, no noise.
+  return (
+    <div className="faint mono" style={xs}>
+      ▸ {e.type}
+    </div>
+  );
+}
+
+/* Compact single-line preview of a tool_use input object. */
+function oneLine(input: unknown): string {
+  const s =
+    typeof input === "string"
+      ? input
+      : (input as Record<string, unknown>)?.command
+        ? String((input as Record<string, unknown>).command)
+        : JSON.stringify(input);
+  return s.length > 80 ? `${s.slice(0, 77)}…` : s;
+}

frontend/components/store.tsx

@@ -17,6 +17,7 @@ import {
   AuthError,
   createClient,
   type Agent,
+  type AgentEvent,
   type ApiError,
   type Approval,
   type Checkmark,
@@ -83,6 +84,9 @@ interface StoreValue {
   log: LogEntry[];
   logOffset: number;
   pageLog: (dir: 1 | -1) => void;
+  /* Headless run event stream for the selected run, oldest-first, appended by cursor
+   * polls (empty for legacy tmux agents). */
+  events: AgentEvent[];
 
   approvals: Approval[];
   hosts: Host[];
@@ -204,6 +208,9 @@ export function DashboardProvider({
   const [checkmarkMissing, setCheckmarkMissing] = useState(false);
   const [log, setLog] = useState<LogEntry[]>([]);
   const [logOffset, setLogOffset] = useState(0);
+  const [events, setEvents] = useState<AgentEvent[]>([]);
+  const eventsRef = useRef<AgentEvent[]>([]);
+  eventsRef.current = events;
   const [approvals, setApprovals] = useState<Approval[]>([]);
   const [hosts, setHosts] = useState<Host[]>([]);
   const [commands, setCommands] = useState<Command[]>([]);
@@ -282,6 +289,17 @@ export function DashboardProvider({
     } catch (e) {
       swallow(e);
     }
+    try {
+      // Cursor poll: only events newer than what we already hold come back.
+      const cur = eventsRef.current;
+      const after = cur.length ? cur[cur.length - 1].id : 0;
+      const fresh = await clientRef.current.api<AgentEvent[]>(
+        `${path}/events?after_id=${after}&limit=500`,
+      );
+      if (fresh.length) setEvents((prev) => [...prev, ...fresh]);
+    } catch (e) {
+      swallow(e);
+    }
   }, []);
 
   const loadApprovals = useCallback(async (projectId: string) => {
@@ -407,6 +425,8 @@ export function DashboardProvider({
       setCheckmark(null);
       setCheckmarkMissing(false);
       setLog([]);
+      setEvents([]);
+      eventsRef.current = [];
       void loadRun(projectId, name);
     },
     [loadRun],
@@ -925,6 +945,7 @@ export function DashboardProvider({
     log,
     logOffset,
     pageLog,
+    events,
     approvals,
     hosts,
     commands,

frontend/lib/api.ts

@@ -30,10 +30,28 @@ export interface Agent {
   working_dir: string;
   status: string;
   role?: string | null;
-  /* Latest tmux pane-tail snapshot from the worker, so the UI can show what a running
+  /* Latest output snapshot from the worker: the tmux pane tail for legacy agents, the
-   * agent is actually doing (and expose one wedged on an interactive prompt). */
+   * latest assistant text for headless runs. For a crashed agent this is the evidence
+   * frame — the last thing the process said. */
   last_output?: string | null;
   output_at?: string | null;
+  /* Headless runner: claude session UUID (null = legacy tmux agent) + supervising worker. */
+  session_id?: string | null;
+  worker_id?: string | null;
+  created_at: string;
+}
+
+/* One persisted stream-json event of a headless run (GET .../events, cursor-paged by id).
+ * `type` mirrors the stream (system/assistant/user/result) plus `worker` (runner notices)
+ * and `raw` (unparseable line kept verbatim). */
+export interface AgentEvent {
+  id: number;
+  agent_id: number;
+  run_id: number;
+  session_id?: string | null;
+  seq: number;
+  type: string;
+  payload?: Record<string, unknown> | null;
   created_at: string;
 }
 

frontend/lib/format.ts

@@ -21,6 +21,7 @@ export function statusTone(status: string | null | undefined): Tone {
     case "blocked":
     case "rejected":
     case "error":
+    case "crashed":
       return "danger";
     case "pending":
     case "queued":

scripts/validate_claude_headless.sh

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+# Phase-1 validation of the real `claude` binary's headless behavior (plan §0).
+# Run MANUALLY inside the control container (needs real claude + credentials); findings
+# go in the phase-1 PR description. Never wired into CI — this probes the real binary.
+#
+#   CLAUDE_BIN=claude ./scripts/validate_claude_headless.sh /tmp/headless-validation
+#
+# Each check prints PASS/FAIL/INFO; the script keeps going so one failure doesn't hide
+# the rest. Items map to the plan's "must validate" list:
+#   1. --verbose requirement with -p --output-format stream-json
+#   2. exact stream-json event shapes for this CLI version
+#   3. --resume from a transcript materialized onto a clean $HOME  (THE LINCHPIN)
+#   4. PreToolUse deny behavior under -p
+#   5. Stop-hook decision:block behavior under -p
+#   6. -p exit-code semantics + credential env var name
+
+set -uo pipefail
+
+CLAUDE_BIN="${CLAUDE_BIN:-claude}"
+WORK="${1:-/tmp/headless-validation}"
+SID="$(python3 -c 'import uuid; print(uuid.uuid4())')"
+
+mkdir -p "$WORK/repo" "$WORK/home-a" "$WORK/home-b"
+cd "$WORK/repo"
+
+note() { printf '\n=== %s ===\n' "$*"; }
+result() { printf -- '--> %s\n' "$*"; }
+
+note "0. version"
+"$CLAUDE_BIN" --version
+
+note "1. does -p --output-format stream-json work WITHOUT --verbose?"
+if HOME="$WORK/home-a" "$CLAUDE_BIN" -p --output-format stream-json \
+    --session-id "$SID" -- 'Reply with the single word: ping' >"$WORK/no-verbose.out" 2>&1; then
+  result "PASS without --verbose (flag optional — keep passing it anyway)"
+else
+  result "FAIL without --verbose (required, as the runner assumes): $(tail -1 "$WORK/no-verbose.out")"
+fi
+
+note "2. event shapes (fresh run, --verbose)"
+SID2="$(python3 -c 'import uuid; print(uuid.uuid4())')"
+HOME="$WORK/home-a" "$CLAUDE_BIN" -p --verbose --output-format stream-json \
+  --session-id "$SID2" -- 'Reply with the single word: pong' \
+  | tee "$WORK/stream.jsonl" \
+  | python3 -c '
+import json, sys
+for line in sys.stdin:
+    line = line.strip()
+    if not line: continue
+    try:
+        e = json.loads(line)
+        keys = ",".join(sorted(e.keys()))
+        print(f"  type={e.get(\"type\")}/{e.get(\"subtype\")} keys=[{keys}]")
+    except Exception:
+        print(f"  UNPARSEABLE: {line[:100]}")
+'
+result "exit code: $? — full stream saved to $WORK/stream.jsonl"
+
+note "3. cross-worker resume: materialize transcript onto a clean HOME (LINCHPIN)"
+MUNGED="$(python3 -c "import sys; print(sys.argv[1].replace('/', '-').replace('.', '-'))" "$WORK/repo")"
+SRC="$WORK/home-a/.claude/projects/$MUNGED"
+DST="$WORK/home-b/.claude/projects/$MUNGED"
+mkdir -p "$DST"
+if cp -r "$SRC/$SID2.jsonl" "$DST/" 2>/dev/null; then
+  [ -d "$SRC/$SID2" ] && cp -r "$SRC/$SID2" "$DST/"
+  if HOME="$WORK/home-b" "$CLAUDE_BIN" -p --verbose --output-format stream-json \
+      --resume "$SID2" -- 'What word did you reply with before? Answer with just that word.' \
+      >"$WORK/resume.jsonl" 2>"$WORK/resume.err"; then
+    if grep -q 'pong' "$WORK/resume.jsonl"; then
+      result "PASS: resume on clean HOME retained context (found 'pong')"
+    else
+      result "PARTIAL: resume ran but context unclear — inspect $WORK/resume.jsonl"
+    fi
+  else
+    result "FAIL: resume on clean HOME errored — $(tail -1 "$WORK/resume.err") (fallback re-injection will be load-bearing)"
+  fi
+else
+  result "FAIL: no transcript found at $SRC — munge algorithm wrong for this version?"
+fi
+
+note "4. PreToolUse deny under -p"
+mkdir -p .claude
+cat > .claude/settings.json <<'EOF'
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "Bash",
+        "hooks": [
+          {"type": "command",
+           "command": "echo '{\"hookSpecificOutput\":{\"hookEventName\":\"PreToolUse\",\"permissionDecision\":\"deny\",\"permissionDecisionReason\":\"validation: always deny\"}}'"}
+        ]
+      }
+    ]
+  },
+  "permissions": {"defaultMode": "acceptEdits", "allow": ["Bash(echo *)"]}
+}
+EOF
+HOME="$WORK/home-a" "$CLAUDE_BIN" -p --verbose --output-format stream-json \
+  --settings .claude/settings.json \
+  -- 'Run the bash command `echo hello` and tell me its output.' \
+  >"$WORK/deny.jsonl" 2>&1
+result "exit=$? — grep the stream for the deny reason:"
+grep -o 'validation: always deny' "$WORK/deny.jsonl" | head -1 \
+  && result "PASS: deny reason surfaced in stream" \
+  || result "INSPECT $WORK/deny.jsonl: deny reason not found"
+
+note "5. Stop hook decision:block under -p"
+cat > .claude/settings.json <<'EOF'
+{
+  "hooks": {
+    "Stop": [
+      {"hooks": [
+        {"type": "command",
+         "command": "if [ -f /tmp/.hv-stop-once ]; then echo '{}'; else touch /tmp/.hv-stop-once; echo '{\"decision\":\"block\",\"reason\":\"validation: say BLOCKED-ONCE then finish\"}'; fi"}
+      ]}
+    ]
+  }
+}
+EOF
+rm -f /tmp/.hv-stop-once
+HOME="$WORK/home-a" "$CLAUDE_BIN" -p --verbose --output-format stream-json \
+  --settings .claude/settings.json -- 'Reply with the word: initial' \
+  >"$WORK/stopblock.jsonl" 2>&1
+result "exit=$? — expect a second turn mentioning BLOCKED-ONCE:"
+grep -o 'BLOCKED-ONCE' "$WORK/stopblock.jsonl" | head -1 \
+  && result "PASS: Stop block re-prompted under -p" \
+  || result "INSPECT $WORK/stopblock.jsonl: no evidence of re-prompt"
+rm -f /tmp/.hv-stop-once
+
+note "6. exit codes"
+HOME="$WORK/home-a" "$CLAUDE_BIN" -p --verbose --output-format stream-json \
+  --resume "$(python3 -c 'import uuid; print(uuid.uuid4())')" -- 'x' \
+  >"$WORK/badresume.out" 2>&1
+result "resume of nonexistent session exit=$? (runner treats nonzero-before-assistant as resume failure)"
+result "INFO: check credential env name with: $CLAUDE_BIN setup-token --help (expect CLAUDE_CODE_OAUTH_TOKEN)"
+
+note "done — artifacts in $WORK"

src/handler/api/routes/agents.py

@@ -14,7 +14,15 @@ from sqlalchemy.exc import IntegrityError
 
 from ...db import repository as repo
 from ..deps import db_conn, require_admin, require_auth
-from ..schemas import AgentIn, AgentOut, CheckmarkOut, CommandOut, LogEntryOut, SpawnIn
+from ..schemas import (
+    AgentEventOut,
+    AgentIn,
+    AgentOut,
+    CheckmarkOut,
+    CommandOut,
+    LogEntryOut,
+    SpawnIn,
+)
 from .common import resolve_agent
 
 router = APIRouter(
@@ -63,13 +71,21 @@ def create_agent(project: str, body: AgentIn, conn: Connection = Depends(db_conn
     dependencies=[Depends(require_admin)],
 )
 def enqueue_spawn(project: str, body: SpawnIn, conn: Connection = Depends(db_conn)) -> dict:
-    """Enqueue a spawn; the worker creates the agent row + tmux session and reports back."""
+    """Enqueue a spawn; the worker creates the agent row + claude process and reports back."""
     _require_project(conn, project)
     if repo.get_agent_by_name(conn, project, body.name) is not None:
         raise HTTPException(
             status.HTTP_409_CONFLICT,
             detail=f"agent '{body.name}' already exists in project '{project}'",
         )
+    if not body.task:
+        # A headless `claude -p` run with no prompt exits immediately having done
+        # nothing. Reject here (400) instead of letting the command fail asynchronously
+        # in the worker.
+        raise HTTPException(
+            status.HTTP_400_BAD_REQUEST,
+            detail="a task is required: the headless runner has no idle-REPL mode",
+        )
     payload = body.model_dump(exclude={"name"}, exclude_none=True)
     return repo.enqueue_command(
         conn,
@@ -114,6 +130,23 @@ def get_checkmark(project: str, name: str, conn: Connection = Depends(db_conn))
     return checkmark
 
 
+@router.get("/{name}/events", response_model=list[AgentEventOut])
+def get_events(
+    project: str,
+    name: str,
+    after_id: int = Query(0, ge=0),
+    limit: int = Query(200, ge=1, le=1000),
+    conn: Connection = Depends(db_conn),
+) -> list[dict]:
+    """The headless run event stream, oldest-first, cursor-paged by row id.
+
+    The UI polls with ``after_id`` = the largest id it has seen, so each poll returns
+    only new events (an empty list for a legacy tmux agent or an idle one).
+    """
+    agent = resolve_agent(conn, project, name)
+    return repo.list_agent_events(conn, agent["id"], after_id=after_id, limit=limit)
+
+
 @router.get("/{name}/log", response_model=list[LogEntryOut])
 def get_log(
     project: str,

src/handler/api/routes/login.py

@@ -43,10 +43,18 @@ def enqueue_login_start(conn: Connection = Depends(db_conn)) -> dict:
     dependencies=[Depends(require_admin)],
 )
 def enqueue_login_submit(body: LoginSubmitIn, conn: Connection = Depends(db_conn)) -> dict:
-    """Feed the pasted authorization code back into the waiting login session."""
+    """Feed the pasted authorization code back into the waiting login session.
+
+    Pinned to the worker that ran ``login_start`` — the live tmux login session exists
+    only in that container, so with multiple workers any other claimant would find
+    nothing to paste into. No prior login_start leaves the pin empty (single-worker
+    deployments behave exactly as before).
+    """
+    started = repo.get_latest_finished_command(conn, "login_start")
     return repo.enqueue_command(
         conn,
         "login_submit",
         payload={"code": body.code},
         requested_by="operator:web",
+        target_worker=started["claimed_by"] if started else None,
     )

src/handler/api/schemas.py

@@ -128,10 +128,14 @@ class AgentOut(BaseModel):
     working_dir: str
     status: str
     role: Role | None = None
-    # Latest tmux pane-tail snapshot (worker poll loop) so the UI can show what a running
+    # Latest output snapshot so the UI can show what a running agent is doing: the tmux
-    # agent is doing — including one wedged on an interactive prompt no one can answer.
+    # pane tail for legacy agents, the latest assistant text for headless runs.
     last_output: str | None = None
     output_at: datetime | None = None
+    # Headless runner: the claude session UUID (null = legacy tmux agent) and the worker
+    # container supervising (or last to supervise) this agent's runs.
+    session_id: str | None = None
+    worker_id: str | None = None
     created_at: datetime
 
 
@@ -306,6 +310,25 @@ class LogEntryOut(BaseModel):
     ci_checked_at: datetime | None = None
 
 
+class AgentEventOut(BaseModel):
+    """One persisted stream-json event of a headless run (the UI's live log panel).
+
+    ``type`` mirrors the stream's top-level type (system/assistant/user/result), plus
+    ``worker`` (runner-generated notices) and ``raw`` (unparseable line, kept verbatim).
+    """
+
+    model_config = ConfigDict(from_attributes=True)
+
+    id: int
+    agent_id: int
+    run_id: int
+    session_id: str | None = None
+    seq: int
+    type: str
+    payload: dict | None = None
+    created_at: datetime
+
+
 class AnswerIn(BaseModel):
     answer: str
     # If omitted, the answer targets the agent's latest open question.

src/handler/api/static/_next/static/chunks/app/layout-b3a2b6165ef8a4d0.js

@@ -1 +0,0 @@
-(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[185],{3156:function(n,e,u){Promise.resolve().then(u.t.bind(u,7960,23))},7960:function(){}},function(n){n.O(0,[587,971,117,744],function(){return n(n.s=3156)}),_N_E=n.O()}]);

src/handler/api/static/_next/static/chunks/app/layout-c871fabc4fd47962.js

@@ -0,0 +1 @@
+(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[185],{2385:function(n,e,u){Promise.resolve().then(u.t.bind(u,7960,23))},7960:function(){}},function(n){n.O(0,[587,971,117,744],function(){return n(n.s=2385)}),_N_E=n.O()}]);

src/handler/api/static/_next/static/chunks/main-app-cb8ae0c3c2bb8b85.js

@@ -1 +1 @@
-(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[744],{6994:function(e,n,t){Promise.resolve().then(t.t.bind(t,2846,23)),Promise.resolve().then(t.t.bind(t,9107,23)),Promise.resolve().then(t.t.bind(t,1060,23)),Promise.resolve().then(t.t.bind(t,4707,23)),Promise.resolve().then(t.t.bind(t,80,23)),Promise.resolve().then(t.t.bind(t,6423,23))}},function(e){var n=function(n){return e(e.s=n)};e.O(0,[971,117],function(){return n(4278),n(6994)}),_N_E=e.O()}]);
+(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[744],{2730:function(e,n,t){Promise.resolve().then(t.t.bind(t,2846,23)),Promise.resolve().then(t.t.bind(t,9107,23)),Promise.resolve().then(t.t.bind(t,1060,23)),Promise.resolve().then(t.t.bind(t,4707,23)),Promise.resolve().then(t.t.bind(t,80,23)),Promise.resolve().then(t.t.bind(t,6423,23))}},function(e){var n=function(n){return e(e.s=n)};e.O(0,[971,117],function(){return n(4278),n(2730)}),_N_E=e.O()}]);

src/handler/api/static/index.html

@@ -1 +1 @@
-<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="stylesheet" href="/_next/static/css/bbd6ee1e7265be74.css" data-precedence="next"/><link rel="preload" as="script" fetchPriority="low" href="/_next/static/chunks/webpack-29957205745576aa.js"/><script src="/_next/static/chunks/fd9d1056-e47440ca368281f5.js" async=""></script><script src="/_next/static/chunks/117-d9d1ed00b6332a85.js" async=""></script><script src="/_next/static/chunks/main-app-8321800782c5a11e.js" async=""></script><script src="/_next/static/chunks/app/page-a511d608db773c67.js" async=""></script><title>Handler · Claude Activity</title><meta name="description" content="Monitor and manage Claude Code agents across projects."/><link rel="icon" href="/icon.svg?bab509b45a421e43" type="image/svg+xml" sizes="any"/><script src="/_next/static/chunks/polyfills-42372ed130431b0a.js" noModule=""></script></head><body><div class="gate"><form class="gate-card"><div class="gate-brand"><span class="logo" style="width:26px;height:26px;border-radius:7px"></span>Claude Monitor</div><p class="muted" style="font-size:var(--text-sm);margin:0">Paste your API token to continue. Management actions require the admin token; read-only views work with the plain auth token.</p><input class="input" type="password" autoComplete="current-password" placeholder="API token" autofocus="" value=""/><button class="btn btn-primary" type="submit">Continue</button></form></div><script src="/_next/static/chunks/webpack-29957205745576aa.js" async=""></script><script>(self.__next_f=self.__next_f||[]).push([0]);self.__next_f.push([2,null])</script><script>self.__next_f.push([1,"1:HL[\"/_next/static/css/bbd6ee1e7265be74.css\",\"style\"]\n"])</script><script>self.__next_f.push([1,"2:I[2846,[],\"\"]\n4:I[9107,[],\"ClientPageRoot\"]\n5:I[9859,[\"931\",\"static/chunks/app/page-a511d608db773c67.js\"],\"default\",1]\n6:I[4707,[],\"\"]\n7:I[6423,[],\"\"]\n9:I[1060,[],\"\"]\na:[]\n0:[\"$\",\"$L2\",null,{\"buildId\":\"tQnVLKX-Dr8H3frHzPrCa\",\"assetPrefix\":\"\",\"urlParts\":[\"\",\"\"],\"initialTree\":[\"\",{\"children\":[\"__PAGE__\",{}]},\"$undefined\",\"$undefined\",true],\"initialSeedData\":[\"\",{\"children\":[\"__PAGE__\",{},[[\"$L3\",[\"$\",\"$L4\",null,{\"props\":{\"params\":{},\"searchParams\":{}},\"Component\":\"$5\"}],null],null],null]},[[[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/bbd6ee1e7265be74.css\",\"precedence\":\"next\",\"crossOrigin\":\"$undefined\"}]],[\"$\",\"html\",null,{\"lang\":\"en\",\"children\":[\"$\",\"body\",null,{\"children\":[\"$\",\"$L6\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\"],\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L7\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":[[\"$\",\"title\",null,{\"children\":\"404: This page could not be found.\"}],[\"$\",\"div\",null,{\"style\":{\"fontFamily\":\"system-ui,\\\"Segoe UI\\\",Roboto,Helvetica,Arial,sans-serif,\\\"Apple Color Emoji\\\",\\\"Segoe UI Emoji\\\"\",\"height\":\"100vh\",\"textAlign\":\"center\",\"display\":\"flex\",\"flexDirection\":\"column\",\"alignItems\":\"center\",\"justifyContent\":\"center\"},\"children\":[\"$\",\"div\",null,{\"children\":[[\"$\",\"style\",null,{\"dangerouslySetInnerHTML\":{\"__html\":\"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}\"}}],[\"$\",\"h1\",null,{\"className\":\"next-error-h1\",\"style\":{\"display\":\"inline-block\",\"margin\":\"0 20px 0 0\",\"padding\":\"0 23px 0 0\",\"fontSize\":24,\"fontWeight\":500,\"verticalAlign\":\"top\",\"lineHeight\":\"49px\"},\"children\":\"404\"}],[\"$\",\"div\",null,{\"style\":{\"display\":\"inline-block\"},\"children\":[\"$\",\"h2\",null,{\"style\":{\"fontSize\":14,\"fontWeight\":400,\"lineHeight\":\"49px\",\"margin\":0},\"children\":\"This page could not be found.\"}]}]]}]}]],\"notFoundStyl"])</script><script>self.__next_f.push([1,"es\":[]}]}]}]],null],null],\"couldBeIntercepted\":false,\"initialHead\":[null,\"$L8\"],\"globalErrorComponent\":\"$9\",\"missingSlots\":\"$Wa\"}]\n"])</script><script>self.__next_f.push([1,"8:[[\"$\",\"meta\",\"0\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1\"}],[\"$\",\"meta\",\"1\",{\"charSet\":\"utf-8\"}],[\"$\",\"title\",\"2\",{\"children\":\"Handler · Claude Activity\"}],[\"$\",\"meta\",\"3\",{\"name\":\"description\",\"content\":\"Monitor and manage Claude Code agents across projects.\"}],[\"$\",\"link\",\"4\",{\"rel\":\"icon\",\"href\":\"/icon.svg?bab509b45a421e43\",\"type\":\"image/svg+xml\",\"sizes\":\"any\"}]]\n3:null\n"])</script></body></html>
+<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="stylesheet" href="/_next/static/css/bbd6ee1e7265be74.css" data-precedence="next"/><link rel="preload" as="script" fetchPriority="low" href="/_next/static/chunks/webpack-29957205745576aa.js"/><script src="/_next/static/chunks/fd9d1056-e47440ca368281f5.js" async=""></script><script src="/_next/static/chunks/117-d9d1ed00b6332a85.js" async=""></script><script src="/_next/static/chunks/main-app-cb8ae0c3c2bb8b85.js" async=""></script><script src="/_next/static/chunks/app/page-0637b8cf149a88ec.js" async=""></script><title>Handler · Claude Activity</title><meta name="description" content="Monitor and manage Claude Code agents across projects."/><link rel="icon" href="/icon.svg?bab509b45a421e43" type="image/svg+xml" sizes="any"/><script src="/_next/static/chunks/polyfills-42372ed130431b0a.js" noModule=""></script></head><body><div class="gate"><form class="gate-card"><div class="gate-brand"><span class="logo" style="width:26px;height:26px;border-radius:7px"></span>Claude Monitor</div><p class="muted" style="font-size:var(--text-sm);margin:0">Paste your API token to continue. Management actions require the admin token; read-only views work with the plain auth token.</p><input class="input" type="password" autoComplete="current-password" placeholder="API token" autofocus="" value=""/><button class="btn btn-primary" type="submit">Continue</button></form></div><script src="/_next/static/chunks/webpack-29957205745576aa.js" async=""></script><script>(self.__next_f=self.__next_f||[]).push([0]);self.__next_f.push([2,null])</script><script>self.__next_f.push([1,"1:HL[\"/_next/static/css/bbd6ee1e7265be74.css\",\"style\"]\n"])</script><script>self.__next_f.push([1,"2:I[2846,[],\"\"]\n4:I[9107,[],\"ClientPageRoot\"]\n5:I[9859,[\"931\",\"static/chunks/app/page-0637b8cf149a88ec.js\"],\"default\",1]\n6:I[4707,[],\"\"]\n7:I[6423,[],\"\"]\n9:I[1060,[],\"\"]\na:[]\n0:[\"$\",\"$L2\",null,{\"buildId\":\"ga21jhjKhYsutf8F3vo_-\",\"assetPrefix\":\"\",\"urlParts\":[\"\",\"\"],\"initialTree\":[\"\",{\"children\":[\"__PAGE__\",{}]},\"$undefined\",\"$undefined\",true],\"initialSeedData\":[\"\",{\"children\":[\"__PAGE__\",{},[[\"$L3\",[\"$\",\"$L4\",null,{\"props\":{\"params\":{},\"searchParams\":{}},\"Component\":\"$5\"}],null],null],null]},[[[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/bbd6ee1e7265be74.css\",\"precedence\":\"next\",\"crossOrigin\":\"$undefined\"}]],[\"$\",\"html\",null,{\"lang\":\"en\",\"children\":[\"$\",\"body\",null,{\"children\":[\"$\",\"$L6\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\"],\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L7\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":[[\"$\",\"title\",null,{\"children\":\"404: This page could not be found.\"}],[\"$\",\"div\",null,{\"style\":{\"fontFamily\":\"system-ui,\\\"Segoe UI\\\",Roboto,Helvetica,Arial,sans-serif,\\\"Apple Color Emoji\\\",\\\"Segoe UI Emoji\\\"\",\"height\":\"100vh\",\"textAlign\":\"center\",\"display\":\"flex\",\"flexDirection\":\"column\",\"alignItems\":\"center\",\"justifyContent\":\"center\"},\"children\":[\"$\",\"div\",null,{\"children\":[[\"$\",\"style\",null,{\"dangerouslySetInnerHTML\":{\"__html\":\"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}\"}}],[\"$\",\"h1\",null,{\"className\":\"next-error-h1\",\"style\":{\"display\":\"inline-block\",\"margin\":\"0 20px 0 0\",\"padding\":\"0 23px 0 0\",\"fontSize\":24,\"fontWeight\":500,\"verticalAlign\":\"top\",\"lineHeight\":\"49px\"},\"children\":\"404\"}],[\"$\",\"div\",null,{\"style\":{\"display\":\"inline-block\"},\"children\":[\"$\",\"h2\",null,{\"style\":{\"fontSize\":14,\"fontWeight\":400,\"lineHeight\":\"49px\",\"margin\":0},\"children\":\"This page could not be found.\"}]}]]}]}]],\"notFoundStyl"])</script><script>self.__next_f.push([1,"es\":[]}]}]}]],null],null],\"couldBeIntercepted\":false,\"initialHead\":[null,\"$L8\"],\"globalErrorComponent\":\"$9\",\"missingSlots\":\"$Wa\"}]\n"])</script><script>self.__next_f.push([1,"8:[[\"$\",\"meta\",\"0\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1\"}],[\"$\",\"meta\",\"1\",{\"charSet\":\"utf-8\"}],[\"$\",\"title\",\"2\",{\"children\":\"Handler · Claude Activity\"}],[\"$\",\"meta\",\"3\",{\"name\":\"description\",\"content\":\"Monitor and manage Claude Code agents across projects.\"}],[\"$\",\"link\",\"4\",{\"rel\":\"icon\",\"href\":\"/icon.svg?bab509b45a421e43\",\"type\":\"image/svg+xml\",\"sizes\":\"any\"}]]\n3:null\n"])</script></body></html>

src/handler/api/static/index.txt

@@ -1,7 +1,7 @@
 2:I[9107,[],"ClientPageRoot"]
-3:I[9859,["931","static/chunks/app/page-a511d608db773c67.js"],"default",1]
+3:I[9859,["931","static/chunks/app/page-0637b8cf149a88ec.js"],"default",1]
 4:I[4707,[],""]
 5:I[6423,[],""]
-0:["tQnVLKX-Dr8H3frHzPrCa",[[["",{"children":["__PAGE__",{}]},"$undefined","$undefined",true],["",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/_next/static/css/bbd6ee1e7265be74.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
+0:["ga21jhjKhYsutf8F3vo_-",[[["",{"children":["__PAGE__",{}]},"$undefined","$undefined",true],["",{"children":["__PAGE__",{},[["$L1",["$","$L2",null,{"props":{"params":{},"searchParams":{}},"Component":"$3"}],null],null],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/_next/static/css/bbd6ee1e7265be74.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","children":["$","body",null,{"children":["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L5",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]],null],null],["$L6",null]]]]
 6:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"Handler · Claude Activity"}],["$","meta","3",{"name":"description","content":"Monitor and manage Claude Code agents across projects."}],["$","link","4",{"rel":"icon","href":"/icon.svg?bab509b45a421e43","type":"image/svg+xml","sizes":"any"}]]
 1:null

src/handler/config.py

@@ -51,6 +51,27 @@ class Settings(BaseSettings):
     forge_bin: str = "forge"
     git_bin: str = "git"
 
+    # ---- Headless runner (claude -p --output-format stream-json): worker-owned
+    # subprocesses streaming events to the DB. Agent runs are always headless; tmux
+    # remains only for the interactive /login flow.
+    # How many concurrent claude runs one worker container supervises; commands that would
+    # start a run are left queued (for another worker) while all slots are busy.
+    max_concurrent_runs: int = 4
+    # Heartbeats older than this many seconds mark a worker dead; the reaper flips its
+    # running runs (and their agents) to ``crashed``.
+    worker_stale_after: float = 60.0
+    # Per-run spend cap passed as ``--max-budget-usd``. 0 disables the flag.
+    run_budget_usd: float = 0.0
+    # Refuse to upload a claude session archive larger than this (a runaway sidecar dir
+    # shouldn't balloon the DB); the run still works, only cross-worker resume degrades.
+    session_archive_max_bytes: int = 32 * 1024 * 1024
+    # settings.json ``permissions.defaultMode`` for headless runs. ``-p`` auto-denies
+    # anything that would prompt interactively, so this plus the allowlist below is what
+    # lets normal work proceed; the PreToolUse/Stop hooks stay the hard gate.
+    headless_permission_mode: str = "acceptEdits"
+    # Comma-separated permission allow rules added to generated settings for headless runs.
+    headless_allowed_tools: str = "Bash(git *),Bash(mise *)"
+
     # The pinned `forge` version (README 3.6 / Phase 2: pin, never float on @latest).
     # When set, spawn verifies the injected forge matches and records a mismatch; when
     # empty the check is skipped. Operators align this with what their base image installs.
@@ -73,6 +94,10 @@ class Settings(BaseSettings):
     def protected_branch_set(self) -> set[str]:
         return {b.strip() for b in self.protected_branches.split(",") if b.strip()}
 
+    @property
+    def headless_allowed_tools_list(self) -> list[str]:
+        return [t.strip() for t in self.headless_allowed_tools.split(",") if t.strip()]
+
     @property
     def cors_origin_list(self) -> list[str]:
         return [o.strip() for o in self.cors_origins.split(",") if o.strip()]

src/handler/control/cli.py

@@ -1,10 +1,11 @@
 """``handler`` CLI — the control layer's write side.
 
-Spawn/list/attach/kill manage agent processes. Phase 2 adds the forge-workflow control
+Spawn/list/kill manage agent runs (headless ``claude -p`` processes supervised by the
-commands: ``approve``/``reject`` (the senior agent records its verdict, which the deploy
+worker). Phase 2 adds the forge-workflow control commands: ``approve``/``reject`` (the
-gate checks), ``poll-ci`` (backfill CI verdicts), and ``forge-init`` (write the role
+senior agent records its verdict, which the deploy gate checks), ``poll-ci`` (backfill
-skills into a managed repo). The DB is the source of truth for what agents exist; tmux is
+CI verdicts), and ``forge-init`` (write the role skills into a managed repo). The DB is
-cross-checked for liveness. All commands are project-namespaced.
+the single source of truth: what agents exist AND whether their runs are live both come
+from it. All commands are project-namespaced.
 """
 
 from __future__ import annotations
@@ -15,7 +16,7 @@ import sys
 
 from ..db import repository as repo
 from ..db.engine import connection
-from . import poller, reposync, skills_gen, spawn, tmux, worker
+from . import poller, reposync, skills_gen, spawn, worker
 
 
 def _cmd_spawn(args: argparse.Namespace) -> int:
@@ -37,38 +38,27 @@ def _cmd_spawn(args: argparse.Namespace) -> int:
         print(f"  role: {args.role}")
     if agent.get("forge_note"):
         print(f"  warning: {agent['forge_note']}", file=sys.stderr)
-    print(f"  tmux session: {tmux.session_name(args.project, args.name)}")
     return 0
 
 
 def _cmd_list(args: argparse.Namespace) -> int:
-    live = set(tmux.list_sessions())
     with connection() as conn:
+        live = {run["agent_id"] for run in repo.list_running_runs(conn)}
         if args.project:
             projects = [args.project] if repo.get_project(conn, args.project) else []
         else:
             projects = [p["id"] for p in repo.list_projects(conn)]
         for project_id in projects:
             for agent in repo.list_agents(conn, project_id):
-                session = tmux.session_name(project_id, agent["name"])
+                alive = "live" if agent["id"] in live else "-"
-                alive = "live" if session in live else "-"
                 role = agent.get("role") or "-"
+                worker_id = agent.get("worker_id") or "-"
                 print(
-                    f"{project_id}/{agent['name']}\t{role}\t{agent['status']}\t{alive}\t{session}"
+                    f"{project_id}/{agent['name']}\t{role}\t{agent['status']}\t{alive}\t{worker_id}"
                 )
     return 0
 
 
-def _cmd_attach(args: argparse.Namespace) -> int:
-    session = tmux.session_name(args.project, args.name)
-    if not tmux.has_session(session):
-        print(f"error: no live session '{session}'", file=sys.stderr)
-        return 1
-    # Replace this process with an interactive tmux attach.
-    os.execvp("tmux", ["tmux", "attach", "-t", session])
-    return 0  # pragma: no cover - execvp does not return
-
-
 def _cmd_kill(args: argparse.Namespace) -> int:
     try:
         spawn.kill(args.project, args.name)
@@ -233,11 +223,6 @@ def build_parser() -> argparse.ArgumentParser:
     p_list.add_argument("--project", help="limit to one project")
     p_list.set_defaults(func=_cmd_list)
 
-    p_attach = sub.add_parser("attach", help="attach to an agent's tmux session")
-    p_attach.add_argument("--project", required=True)
-    p_attach.add_argument("--name", required=True)
-    p_attach.set_defaults(func=_cmd_attach)
-
     p_kill = sub.add_parser("kill", help="kill an agent's session")
     p_kill.add_argument("--project", required=True)
     p_kill.add_argument("--name", required=True)

src/handler/control/credsync.py

@@ -0,0 +1,171 @@
+"""Distribute claude's OAuth credentials to every worker through the database.
+
+The interactive ``/login`` flow (see :mod:`~handler.control.login`) completes on exactly
+one worker container and writes credentials under that container's ``$HOME``. With
+multiple headless workers, the others need those files too — and the deployment rule is
+"no shared filesystems, the DB is the single source of truth". So:
+
+- ``upload()`` bundles the credential files, encrypts the bundle with the existing
+  Fernet secret store (``HANDLER_SECRET_KEY``), and upserts it into ``runtime_secrets``.
+  Called after a confirmed login, and after a run if claude refreshed the token on disk.
+- ``refresh()`` (workers, at startup and periodically) materializes the bundle locally
+  when the DB copy is newer than what this worker last saw. ``~/.claude.json`` is merged
+  (only credential-ish top-level keys are taken) so a worker's own onboarding/trust
+  state written by ``claude_config.ensure_onboarded`` survives; the pure credential file
+  is written verbatim.
+
+Everything is best-effort and silently off when no ``HANDLER_SECRET_KEY`` is configured
+(single-worker deployments work exactly as before).
+"""
+
+from __future__ import annotations
+
+import json
+import os
+
+from .. import secretstore
+from ..db import repository as repo
+from ..db.engine import connection
+
+SECRET_KEY = "claude_credentials"
+
+# ~/.claude.json keys that belong to the *account*, not this machine's UI/trust state.
+_ACCOUNT_KEYS = ("oauthAccount", "userID", "organization", "account")
+
+
+def _credential_files() -> dict[str, str]:
+    """Relative path -> absolute path of every claude credential file we bundle."""
+    home = os.path.expanduser("~")
+    return {
+        ".claude.json": os.path.join(home, ".claude.json"),
+        ".claude/.credentials.json": os.path.join(home, ".claude", ".credentials.json"),
+    }
+
+
+def fingerprint() -> tuple:
+    """(path, mtime_ns, size) of the OAuth token file — cheap change detection.
+
+    Deliberately only ``.claude/.credentials.json``: claude touches ``~/.claude.json``
+    on every run (project entries, UI state), and treating those as "new credentials"
+    would ping-pong uploads between workers forever. A login that only rewrites
+    ``.claude.json`` is still published — the login flow calls :func:`upload` directly.
+    """
+    path = _credential_files()[".claude/.credentials.json"]
+    try:
+        st = os.stat(path)
+    except OSError:
+        return ()
+    return ((path, st.st_mtime_ns, st.st_size),)
+
+
+def upload() -> bool:
+    """Encrypt the local credential files into ``runtime_secrets``. Returns whether a
+    bundle was stored (False when disabled or there is nothing to store)."""
+    if not secretstore.enabled():
+        return False
+    bundle: dict[str, str] = {}
+    for rel, path in _credential_files().items():
+        try:
+            with open(path) as fh:
+                bundle[rel] = fh.read()
+        except OSError:
+            continue
+    if not bundle:
+        return False
+    value_enc = secretstore.encrypt(json.dumps(bundle))
+    with connection() as conn:
+        repo.upsert_runtime_secret(conn, SECRET_KEY, value_enc)
+    return True
+
+
+def _merge_claude_json(path: str, incoming: str) -> None:
+    """Take the account keys from ``incoming`` into ``path``, preserving local state.
+
+    A worker's ``~/.claude.json`` also carries per-directory trust + onboarding flags for
+    *its* clones; clobbering those would re-wedge spawns on the trust dialog.
+    """
+    try:
+        new_data = json.loads(incoming)
+    except (ValueError, TypeError):
+        return
+    if not isinstance(new_data, dict):
+        return
+    current: dict = {}
+    if os.path.exists(path):
+        try:
+            with open(path) as fh:
+                current = json.load(fh)
+        except (OSError, ValueError):
+            current = {}
+        if not isinstance(current, dict):
+            current = {}
+    if not current:
+        merged = new_data
+    else:
+        merged = current
+        for key in _ACCOUNT_KEYS:
+            if key in new_data:
+                merged[key] = new_data[key]
+    os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
+    tmp = f"{path}.credsync.tmp"
+    with open(tmp, "w") as fh:
+        json.dump(merged, fh, indent=2)
+    os.replace(tmp, path)
+
+
+class _State:
+    """Per-process sync cursor: the DB updated_at we last materialized, and the local
+    file fingerprint after the last upload/materialize (to detect claude refreshing
+    the token mid-run so the next refresh() re-uploads)."""
+
+    def __init__(self) -> None:
+        self.seen_updated_at = None
+        self.last_fingerprint: tuple | None = None
+
+
+_state = _State()
+
+
+def refresh() -> str | None:
+    """One sync pass; returns "materialized", "uploaded", or None (no-op).
+
+    - A DB bundle newer than what we've seen wins: materialize it locally.
+    - Otherwise, local credential files that changed since our last pass (a login on
+      this worker, or claude refreshing a token during a run) get uploaded.
+    """
+    if not secretstore.enabled():
+        return None
+    with connection() as conn:
+        row = repo.get_runtime_secret(conn, SECRET_KEY)
+
+    if row is not None and row["updated_at"] != _state.seen_updated_at:
+        try:
+            bundle = json.loads(secretstore.decrypt(row["value_enc"]))
+        except (secretstore.SecretStoreError, ValueError):
+            return None
+        files = _credential_files()
+        for rel, content in bundle.items():
+            path = files.get(rel)
+            if path is None:
+                continue  # never write outside the known credential set
+            if rel == ".claude.json":
+                _merge_claude_json(path, content)
+            else:
+                os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
+                tmp = f"{path}.credsync.tmp"
+                with open(tmp, "w") as fh:
+                    fh.write(content)
+                os.replace(tmp, path)
+        _state.seen_updated_at = row["updated_at"]
+        _state.last_fingerprint = fingerprint()
+        return "materialized"
+
+    current = fingerprint()
+    if current and current != _state.last_fingerprint:
+        if upload():
+            _state.last_fingerprint = current
+            with connection() as conn:
+                stored = repo.get_runtime_secret(conn, SECRET_KEY)
+            _state.seen_updated_at = stored["updated_at"] if stored else None
+            return "uploaded"
+    return None

src/handler/control/headless.py

@@ -0,0 +1,385 @@
+"""The headless runner: worker-owned ``claude -p`` subprocesses streaming to the DB.
+
+This is the tmux replacement seam for agent *runs* (tmux stays only for the interactive
+``/login`` flow). Each invocation is ``claude -p --output-format stream-json`` with a
+pre-assigned ``--session-id``; a :class:`RunSupervisor` owns the child process, appends
+every stdout JSON line to ``agent_events`` as it arrives, derives ``agents.last_output``
+from the latest assistant text (so the existing UI keeps working), and reconciles the
+agent's status from the *process* — exit code and EOF are positive liveness, replacing
+the old pane scraping that could neither see a dead process nor populate the log.
+
+Resume continuity without shared files: claude persists its session under
+``~/.claude/projects/<munged-cwd>/``; the supervisor tars that into ``session_archives``
+(periodically and at exit), and whichever worker later claims a resume materializes the
+archive at the same munged path before running ``claude -p --resume``. Workers therefore
+need the same ``projects_root`` layout — a deployment invariant — but share nothing.
+
+``launch()`` is the function ``spawn``/``worker`` call and tests mock (via the
+``claude_bin`` setting pointing at a fake, same pattern as the tmux fakes).
+"""
+
+from __future__ import annotations
+
+import io
+import json
+import os
+import subprocess
+import tarfile
+import tempfile
+import threading
+import time
+import uuid
+from pathlib import Path
+
+from ..config import get_settings
+from ..db import repository as repo
+from ..db.engine import connection
+
+# Stream types we recognize from ``--output-format stream-json``; anything else (or an
+# unparseable line) is stored as-is so no output is ever dropped. ``worker`` is our own:
+# runner-generated notices (crashes, archive failures, degraded resumes).
+_KNOWN_TYPES = ("system", "assistant", "user", "result", "hook")
+
+# How long a SIGTERM'd claude gets to die before SIGKILL.
+_TERM_GRACE_SECONDS = 10.0
+
+
+def munged_project_dir(working_dir: str) -> str:
+    """The directory name claude uses for a cwd under ``~/.claude/projects/``.
+
+    Claude munges the absolute path by mapping ``/`` and ``.`` to ``-`` (verified against
+    real session state: ``/root/handler`` → ``-root-handler``,
+    ``/root/Talos/.claude/worktrees/x`` → ``-root-Talos--claude-worktrees-x``).
+    """
+    return working_dir.replace("/", "-").replace(".", "-")
+
+
+def session_dir(working_dir: str) -> Path:
+    """Where claude persists sessions for ``working_dir`` (under the *current* $HOME)."""
+    return Path(os.path.expanduser("~")) / ".claude" / "projects" / munged_project_dir(working_dir)
+
+
+def build_spawn_argv(task: str, settings_path: str, session_id: str) -> list[str]:
+    """The headless spawn invocation. ``--verbose`` is required with stream-json in
+    print mode; ``--session-id`` pre-assigns the UUID so the session is addressable
+    (and archivable) from the first event."""
+    s = get_settings()
+    argv = [
+        s.claude_bin, "-p", "--verbose",
+        "--output-format", "stream-json",
+        "--session-id", session_id,
+        "--settings", settings_path,
+    ]
+    if s.run_budget_usd > 0:
+        argv += ["--max-budget-usd", str(s.run_budget_usd)]
+    argv += ["--", task]
+    return argv
+
+
+def build_resume_argv(session_id: str, answer: str, settings_path: str) -> list[str]:
+    """The headless resume invocation — a brand-new process continuing ``session_id``."""
+    s = get_settings()
+    argv = [
+        s.claude_bin, "-p", "--verbose",
+        "--output-format", "stream-json",
+        "--resume", session_id,
+        "--settings", settings_path,
+    ]
+    if s.run_budget_usd > 0:
+        argv += ["--max-budget-usd", str(s.run_budget_usd)]
+    argv += ["--", answer]
+    return argv
+
+
+def parse_stream_line(line: str) -> tuple[str, dict]:
+    """One stdout line → ``(event_type, payload)``. Never raises: malformed JSON or an
+    unrecognized shape comes back as ``("raw", {"line": ...})`` so the event log keeps
+    everything the process said, even across CLI format drift."""
+    text = line.strip()
+    if not text:
+        return "raw", {"line": line}
+    try:
+        payload = json.loads(text)
+    except (ValueError, TypeError):
+        return "raw", {"line": line}
+    if not isinstance(payload, dict):
+        return "raw", {"line": line}
+    etype = payload.get("type")
+    if not isinstance(etype, str) or not etype:
+        return "raw", payload
+    if etype not in _KNOWN_TYPES:
+        # Future/unknown top-level types still store under their own name — the UI
+        # ignores what it doesn't know, but nothing is lost.
+        return etype, payload
+    return etype, payload
+
+
+def assistant_text(payload: dict) -> str | None:
+    """The concatenated text blocks of an ``assistant`` event, or None when it carries
+    none (e.g. a pure tool_use turn). Feeds ``agents.last_output``."""
+    message = payload.get("message")
+    if not isinstance(message, dict):
+        return None
+    content = message.get("content")
+    if isinstance(content, str):
+        return content or None
+    if not isinstance(content, list):
+        return None
+    parts = [
+        block.get("text", "")
+        for block in content
+        if isinstance(block, dict) and block.get("type") == "text"
+    ]
+    text = "\n".join(p for p in parts if p)
+    return text or None
+
+
+def archive_session(working_dir: str, session_id: str, max_bytes: int | None = None) -> bytes | None:
+    """Tar.gz the session transcript + sidecar dir, or None when nothing exists yet or
+    the result would exceed ``max_bytes`` (the caller records a worker event; the run
+    itself is unaffected — only cross-worker resume degrades)."""
+    base = session_dir(working_dir)
+    jsonl = base / f"{session_id}.jsonl"
+    sidecar = base / session_id
+    if not jsonl.exists() and not sidecar.exists():
+        return None
+    buf = io.BytesIO()
+    with tarfile.open(fileobj=buf, mode="w:gz") as tar:
+        if jsonl.exists():
+            tar.add(jsonl, arcname=jsonl.name)
+        if sidecar.is_dir():
+            tar.add(sidecar, arcname=sidecar.name)
+    data = buf.getvalue()
+    limit = max_bytes if max_bytes is not None else get_settings().session_archive_max_bytes
+    if limit and len(data) > limit:
+        return None
+    return data
+
+
+def materialize_session(working_dir: str, archive: bytes) -> None:
+    """Unpack a session archive where claude will look for it on ``--resume``.
+
+    ``filter="data"`` rejects path traversal and special members — the archive came from
+    our own DB, but a defense-in-depth default costs nothing.
+    """
+    base = session_dir(working_dir)
+    base.mkdir(parents=True, exist_ok=True)
+    with tarfile.open(fileobj=io.BytesIO(archive), mode="r:gz") as tar:
+        tar.extractall(base, filter="data")
+
+
+class RunSupervisor:
+    """Owns one headless claude subprocess for its whole life.
+
+    A reader thread pumps stdout lines into ``agent_events``; the supervisor thread
+    watches the process, polls ``cancel_requested`` (cross-worker kill), uploads the
+    session archive periodically, and on exit reconciles run + agent status. Threads are
+    daemons: if the whole worker dies, the reaper — not us — settles the record.
+    """
+
+    def __init__(
+        self,
+        agent: dict,
+        run: dict,
+        argv: list[str],
+        cwd: str,
+        env: dict[str, str],
+        *,
+        cancel_poll: float = 5.0,
+        archive_interval: float = 60.0,
+        on_exit=None,
+    ) -> None:
+        self.agent = agent
+        self.run = run
+        self.argv = argv
+        self.cwd = cwd
+        self.env = env
+        self.cancel_poll = cancel_poll
+        self.archive_interval = archive_interval
+        self.on_exit = on_exit  # worker's slot-release callback
+        self._seq = 0
+        self._result_payload: dict | None = None
+        self._canceled = False
+        self.thread: threading.Thread | None = None
+
+    def start(self) -> None:
+        self.thread = threading.Thread(
+            target=self._supervise, name=f"run-{self.run['id']}", daemon=True
+        )
+        self.thread.start()
+
+    # ---------------------------------------------------------------- internals
+
+    def _insert_event(self, etype: str, payload: dict) -> None:
+        self._seq += 1
+        with connection() as conn:
+            repo.insert_agent_event(
+                conn,
+                self.agent["id"],
+                self.run["id"],
+                seq=self._seq,
+                type=etype,
+                payload=payload,
+                session_id=self.run["session_id"],
+            )
+
+    def _pump_stdout(self, stream) -> None:
+        for line in stream:
+            etype, payload = parse_stream_line(line)
+            try:
+                self._insert_event(etype, payload)
+                if etype == "result":
+                    self._result_payload = payload
+                elif etype == "assistant":
+                    text = assistant_text(payload)
+                    if text:
+                        with connection() as conn:
+                            repo.update_agent_output(conn, self.agent["id"], text)
+            except Exception:  # noqa: BLE001 - a DB hiccup must not sever the pipe
+                continue
+
+    def _upload_archive(self) -> None:
+        try:
+            data = archive_session(self.cwd, self.run["session_id"])
+            if data is None:
+                return
+            with connection() as conn:
+                repo.upsert_session_archive(
+                    conn, self.agent["id"], self.run["session_id"], data
+                )
+        except Exception as exc:  # noqa: BLE001 - archiving is best-effort
+            try:
+                self._insert_event(
+                    "worker", {"notice": "session archive failed", "error": str(exc)}
+                )
+            except Exception:  # noqa: BLE001
+                pass
+
+    def _terminate(self, proc: subprocess.Popen) -> None:
+        self._canceled = True
+        proc.terminate()
+        try:
+            proc.wait(timeout=_TERM_GRACE_SECONDS)
+        except subprocess.TimeoutExpired:
+            proc.kill()
+
+    def _supervise(self) -> None:
+        stderr_file = tempfile.TemporaryFile()
+        try:
+            proc = subprocess.Popen(
+                self.argv,
+                cwd=self.cwd,
+                env={**os.environ, **self.env},
+                stdin=subprocess.DEVNULL,
+                stdout=subprocess.PIPE,
+                stderr=stderr_file,
+                text=True,
+            )
+        except OSError as exc:
+            stderr_file.close()
+            self._settle(exit_code=None, stderr_tail=f"failed to launch claude: {exc}")
+            return
+        reader = threading.Thread(
+            target=self._pump_stdout, args=(proc.stdout,), daemon=True
+        )
+        reader.start()
+        last_archive = time.monotonic()
+        next_cancel_check = time.monotonic() + self.cancel_poll
+        while proc.poll() is None:
+            time.sleep(0.2)
+            now = time.monotonic()
+            if now >= next_cancel_check:
+                next_cancel_check = now + self.cancel_poll
+                with connection() as conn:
+                    if repo.get_cancel_requested(conn, self.run["id"]):
+                        self._terminate(proc)
+                        break
+            if self.archive_interval > 0 and now - last_archive >= self.archive_interval:
+                self._upload_archive()
+                last_archive = now
+        proc.wait()
+        reader.join(timeout=30.0)
+        stderr_file.seek(0)
+        stderr_tail = stderr_file.read()[-4000:].decode("utf-8", "replace")
+        stderr_file.close()
+        self._settle(exit_code=proc.returncode, stderr_tail=stderr_tail)
+
+    def _settle(self, exit_code: int | None, stderr_tail: str) -> None:
+        """Reconcile run + agent status once the process is gone, then final-archive."""
+        result = self._result_payload
+        clean = (
+            exit_code == 0
+            and result is not None
+            and not result.get("is_error", False)
+        )
+        if self._canceled:
+            run_status = "canceled"
+        elif clean:
+            run_status = "completed"
+        else:
+            run_status = "failed"
+        try:
+            with connection() as conn:
+                finished = repo.finish_run(
+                    conn, self.run["id"], run_status, exit_code=exit_code, result=result
+                )
+                # Hooks are the authority on agent status — they ran inside the run and
+                # may have set paused_for_input/blocked/done already. Only an agent still
+                # marked ``working`` needs the process's verdict.
+                agent = repo.get_agent_by_id(conn, self.agent["id"])
+                if finished and agent is not None and agent["status"] == "working":
+                    if self._canceled:
+                        repo.set_agent_status(conn, self.agent["id"], "done")
+                    elif clean:
+                        repo.set_agent_status(conn, self.agent["id"], "done")
+                    else:
+                        repo.set_agent_status(conn, self.agent["id"], "blocked")
+            if not clean and not self._canceled:
+                detail = {"notice": "run failed", "exit_code": exit_code}
+                if stderr_tail.strip():
+                    detail["stderr_tail"] = stderr_tail
+                self._insert_event("worker", detail)
+        except Exception:  # noqa: BLE001 - never let bookkeeping raise out of the thread
+            pass
+        self._upload_archive()
+        if self.on_exit is not None:
+            try:
+                self.on_exit(self)
+            except Exception:  # noqa: BLE001
+                pass
+
+
+def launch(
+    agent: dict,
+    *,
+    kind: str,
+    prompt: str,
+    settings_path: str,
+    env: dict[str, str],
+    worker_id: str,
+    on_exit=None,
+) -> dict:
+    """Start a headless run for ``agent`` and return its ``agent_runs`` row.
+
+    ``kind`` is ``spawn`` (fresh session, new UUID) or ``resume`` (materialize the stored
+    archive, continue the agent's existing session). Fire-and-forget from the caller's
+    perspective — the returned run row is already ``running`` and a daemon supervisor
+    owns the process from here.
+    """
+    working_dir = agent["working_dir"]
+    if kind == "spawn":
+        session_id = str(uuid.uuid4())
+        argv = build_spawn_argv(prompt, settings_path, session_id)
+    else:
+        session_id = agent.get("session_id")
+        if not session_id:
+            raise ValueError(f"agent '{agent['name']}' has no session to resume")
+        argv = build_resume_argv(session_id, prompt, settings_path)
+    with connection() as conn:
+        run = repo.create_run(conn, agent["id"], session_id, worker_id, kind)
+        repo.set_agent_session(conn, agent["id"], session_id, worker_id)
+    supervisor = RunSupervisor(
+        agent, run, argv, cwd=working_dir, env=env, on_exit=on_exit
+    )
+    supervisor.start()
+    return run

src/handler/control/settings_gen.py

@@ -12,6 +12,8 @@ import json
 import os
 import sys
 
+from ..config import get_settings
+
 
 def _hook_command(event: str) -> str:
     # Use the exact interpreter the control layer runs under, so the hook resolves the
@@ -20,7 +22,7 @@ def _hook_command(event: str) -> str:
 
 
 def build_settings() -> dict:
-    return {
+    settings = {
         "hooks": {
             "Stop": [{"hooks": [{"type": "command", "command": _hook_command("stop")}]}],
             "SessionEnd": [
@@ -37,6 +39,16 @@ def build_settings() -> dict:
             ],
         }
     }
+    # ``claude -p`` never prompts — anything that would ask for permission is
+    # auto-denied. The allowlist is therefore what lets normal work (git, mise, the
+    # project's own tooling) proceed; the PreToolUse/Stop hooks above remain the hard
+    # gate either way, since a hook deny overrides any allow.
+    s = get_settings()
+    settings["permissions"] = {
+        "defaultMode": s.headless_permission_mode,
+        "allow": s.headless_allowed_tools_list,
+    }
+    return settings
 
 
 def write_settings(working_dir: str) -> str:

src/handler/control/spawn.py

@@ -15,14 +15,13 @@ from ..config import get_settings
 from ..db import repository as repo
 from ..db.engine import connection
 from . import (
-    claude_config,
     credentials,
     forge,
     gitops,
+    headless,
     mise,
     reposync,
     settings_gen,
-    tmux,
     worktree,
 )
 
@@ -45,18 +44,6 @@ def require_test_task(working_dir: str) -> None:
         )
 
 
-def _claude_command(task: str | None, settings_path: str) -> str:
-    claude = get_settings().claude_bin
-    argv = [claude, "--settings", settings_path]
-    if task:
-        argv.append(_shell_quote(task))
-    return " ".join(argv)
-
-
-def _shell_quote(value: str) -> str:
-    return "'" + value.replace("'", "'\\''") + "'"
-
-
 def _install_git_credentials(
     working_dir: str, git_remote: str | None, conn=None
 ) -> None:
@@ -84,6 +71,7 @@ def spawn(
     role: str | None = None,
     require_tests: bool = True,
     mise_init: bool = False,
+    worker_id: str | None = None,
 ) -> dict:
     """Create and launch an agent. Returns the agent row.
 
@@ -91,8 +79,13 @@ def spawn(
     it off, because a project with no ``.mise.toml`` yet is exactly what it exists to fix.
     ``mise_init`` marks the launched agent (via ``HANDLER_MISE_INIT``) so its hooks enforce
     the bootstrap contract — create the test task, commit, and push — instead of the normal
-    test gate.
+    test gate. ``worker_id`` identifies the calling worker container (headless runs record
+    it on the run row; the CLI defaults to a pid-scoped id).
     """
+    if not task:
+        # ``claude -p`` has no idle-REPL mode — an empty prompt would exit immediately
+        # having done nothing, so a task is a hard requirement.
+        raise SpawnError("an agent requires a task (headless claude has no idle mode)")
     sync_note = None
     with connection() as conn:
         project = repo.get_project(conn, project_id)
@@ -147,13 +140,43 @@ def spawn(
         )
 
     settings_path = settings_gen.write_settings(working_dir)
+    env = _agent_env(project, agent, token, role=role, mise_init=mise_init)
 
+    # Verify the pinned forge version, if one is configured. Non-fatal: a version drift
+    # is recorded as a warning rather than blocking the spawn, since not every agent
+    # touches forge and the base image is the real pin (README 3.6, Phase 2).
+    forge_note = _check_forge_version(working_dir)
+
+    headless.launch(
+        agent,
+        kind="spawn",
+        prompt=task,
+        settings_path=settings_path,
+        env=env,
+        worker_id=worker_id or f"cli-{os.getpid()}",
+    )
+    agent = {**agent, "forge_note": forge_note, "sync_note": sync_note}
+    return agent
+
+
+def _agent_env(
+    project: dict,
+    agent: dict,
+    token: str | None,
+    *,
+    role: str | None = None,
+    mise_init: bool = False,
+) -> dict[str, str]:
+    """The environment an agent process (and therefore its hooks) runs with: identity,
+    ``DATABASE_URL``, and resolved forge/git credentials. Shared by spawn and the
+    headless resume path (a resume is a brand-new process needing the same env)."""
     env = {
-        "HANDLER_PROJECT_ID": project_id,
+        "HANDLER_PROJECT_ID": project["id"],
-        "HANDLER_AGENT_NAME": name,
+        "HANDLER_AGENT_NAME": agent["name"],
         "HANDLER_AGENT_ID": str(agent["id"]),
         "DATABASE_URL": get_settings().database_url,
     }
+    role = role or agent.get("role")
     if role:
         env["HANDLER_AGENT_ROLE"] = role
     if mise_init:
@@ -164,28 +187,13 @@ def spawn(
     with connection() as conn:
         env.update(credentials.credential_env(token, project.get("git_remote"), conn))
         if token:
-            _install_git_credentials(working_dir, project.get("git_remote"), conn)
+            _install_git_credentials(agent["working_dir"], project.get("git_remote"), conn)
         # SSH remotes: pin the agent's git to the server's deploy key, when one is stored.
         try:
             env.update(reposync.ssh_env(project.get("git_remote"), conn))
         except reposync.SyncError as exc:
             raise SpawnError(str(exc)) from exc
-
+    return env
-    # Verify the pinned forge version, if one is configured. Non-fatal: a version drift
-    # is recorded as a warning rather than blocking the spawn, since not every agent
-    # touches forge and the base image is the real pin (README 3.6, Phase 2).
-    forge_note = _check_forge_version(working_dir)
-
-    # Mark Claude Code onboarding complete + trust the working dir before launching, so the
-    # detached agent boots straight to the REPL instead of wedging on the first-run theme
-    # picker / trust prompt with no human at the tmux TTY to answer it.
-    claude_config.ensure_onboarded(working_dir)
-
-    session = tmux.session_name(project_id, name)
-    command = _claude_command(task, settings_path)
-    tmux.new_session(session, cwd=working_dir, command=command, env=env)
-    agent = {**agent, "forge_note": forge_note, "sync_note": sync_note}
-    return agent
 
 
 def _check_forge_version(working_dir: str) -> str | None:
@@ -199,24 +207,120 @@ def _check_forge_version(working_dir: str) -> str | None:
 
 
 def kill(project_id: str, name: str) -> None:
+    """Stop an agent: flag its running run for cancel and mark the row done.
+
+    The owning worker's supervisor polls the cancel flag and SIGTERMs its own child
+    (cross-worker safe — nobody signals a process they don't own). No running run means
+    the process is already gone; the status update is all that's left to do.
+    """
     with connection() as conn:
         agent = repo.get_agent_by_name(conn, project_id, name)
         if agent is None:
             raise SpawnError(f"agent '{name}' not found in project '{project_id}'")
-        session = tmux.session_name(project_id, name)
+        run = repo.get_latest_run(conn, agent["id"])
-        if tmux.has_session(session):
+        if run is not None and run["status"] == "running":
-            tmux.kill_session(session)
+            repo.request_run_cancel(conn, run["id"])
         repo.set_agent_status(conn, agent["id"], "done")
 
 
-def resume(agent: dict, answer: str) -> tuple[bool, str]:
+def resume(agent: dict, answer: str, worker_id: str | None = None) -> tuple[bool, str]:
-    """Feed an operator's answer back to a live agent.
+    """Feed an operator's answer back to an agent as a new ``claude -p --resume`` run.
 
-    The seam the API's ``/resume`` route calls (and the one tests mock). Sends the
+    The seam the API's ``/resume`` route calls (and the one tests mock). The session
-    answer into the agent's tmux session so the waiting ``claude`` process receives it.
+    transcript is materialized from the DB archive first, so ANY worker can serve the
+    resume. Refuses while a run is still live (two concurrent processes on one session
+    would corrupt it). When no transcript survives anywhere — the owning worker died
+    before its first archive, or the row predates the headless runner — falls back to a
+    *fresh* session whose prompt re-injects context from the DB (checkmark + open
+    question + answer), recorded as a ``worker`` event so the UI shows the degraded
+    continuity.
     """
-    session = tmux.session_name(agent["project_id"], agent["name"])
+    worker_id = worker_id or f"cli-{os.getpid()}"
-    if not tmux.has_session(session):
+    with connection() as conn:
-        return False, f"no live session '{session}' to resume"
+        run = repo.get_latest_run(conn, agent["id"])
-    tmux.send_keys(session, answer)
+        if run is not None and run["status"] == "running":
-    return True, f"answer delivered to session '{session}'"
+            return False, "agent already has a live run; wait for it to finish or kill it"
+        project = repo.get_project(conn, agent["project_id"])
+        archive = repo.get_session_archive(conn, agent["id"])
+    if project is None:
+        return False, f"project '{agent['project_id']}' not registered"
+
+    working_dir = agent["working_dir"]
+    settings_path = settings_gen.write_settings(working_dir)
+    try:
+        token = None
+        with connection() as conn:
+            token = credentials.resolve_for_project(project, conn)
+    except credentials.CredentialError as exc:
+        return False, str(exc)
+    env = _agent_env(project, agent, token)
+
+    if not agent.get("session_id"):
+        # Pre-headless agent row (or a spawn that never launched): nothing to --resume.
+        return _resume_reinjected(agent, answer, settings_path, env, worker_id)
+
+    transcript = headless.session_dir(working_dir) / f"{agent['session_id']}.jsonl"
+    if archive is not None:
+        try:
+            headless.materialize_session(working_dir, bytes(archive["archive"]))
+        except (OSError, ValueError) as exc:
+            return False, f"could not materialize session archive: {exc}"
+    elif not transcript.exists():
+        return _resume_reinjected(agent, answer, settings_path, env, worker_id)
+
+    run = headless.launch(
+        agent,
+        kind="resume",
+        prompt=answer,
+        settings_path=settings_path,
+        env=env,
+        worker_id=worker_id,
+    )
+    return True, f"headless resume run {run['id']} started for session {agent['session_id']}"
+
+
+def _resume_reinjected(
+    agent: dict, answer: str, settings_path: str, env: dict, worker_id: str
+) -> tuple[bool, str]:
+    """Degraded resume: no transcript anywhere, so start a fresh session with the
+    context rebuilt from the DB. Continuity is approximate — say so in the event log."""
+    with connection() as conn:
+        checkmark = repo.get_checkmark(conn, agent["id"])
+        recent = repo.get_log(conn, agent["id"], limit=5)
+    parts = [
+        "You are resuming work you started in an earlier session whose transcript is "
+        "unavailable. Reconstruct context from your checkpoint below, then continue.",
+    ]
+    if checkmark:
+        if checkmark.get("where_it_stopped"):
+            parts.append(f"Where you stopped: {checkmark['where_it_stopped']}")
+        if checkmark.get("next_steps"):
+            parts.append(f"Planned next steps: {checkmark['next_steps']}")
+        if checkmark.get("open_question"):
+            parts.append(f"You had asked: {checkmark['open_question']}")
+    for entry in reversed(recent):
+        if entry.get("summary"):
+            parts.append(f"Earlier log: {entry['summary']}")
+    parts.append(f"The operator's answer/instruction: {answer}")
+    run = headless.launch(
+        agent,
+        kind="spawn",  # a genuinely new session (new UUID) — --resume has nothing to load
+        prompt="\n\n".join(parts),
+        settings_path=settings_path,
+        env=env,
+        worker_id=worker_id,
+    )
+    with connection() as conn:
+        repo.insert_agent_event(
+            conn,
+            agent["id"],
+            run["id"],
+            seq=0,
+            type="worker",
+            payload={
+                "notice": "resume without transcript — context re-injected from DB",
+                "previous_session_id": agent["session_id"],
+            },
+            session_id=run["session_id"],
+        )
+    return True, f"transcript unavailable; started fresh run {run['id']} with re-injected context"

src/handler/control/tmux.py

@@ -1,7 +1,8 @@
-"""Thin tmux wrapper — the single mock seam for spawning.
+"""Thin tmux wrapper — now used ONLY by the interactive ``/login`` flow.
 
-Every tmux/claude invocation goes through these functions so tests can substitute a
+Agent runs are headless (``control.headless``); the one thing that still genuinely
-fake and never touch a real tmux server or ``claude`` binary.
+needs a TTY is driving claude's ``/login`` OAuth screens. Everything here goes through
+subprocess so the login tests can substitute a fake and never touch a real tmux server.
 """
 
 from __future__ import annotations
@@ -11,14 +12,6 @@ import subprocess
 from ..config import get_settings
 
 
-def session_name(project_id: str, agent_name: str) -> str:
-    """``project__agent`` with tmux-illegal characters sanitized (README 3.4)."""
-    safe = f"{project_id}__{agent_name}"
-    for ch in (".", ":", " "):
-        safe = safe.replace(ch, "-")
-    return safe
-
-
 def new_session(
     name: str,
     cwd: str,
@@ -57,18 +50,6 @@ def has_session(name: str) -> bool:
     return result.returncode == 0
 
 
-def list_sessions() -> list[str]:
-    tmux = get_settings().tmux_bin
-    result = subprocess.run(
-        [tmux, "list-sessions", "-F", "#{session_name}"],
-        capture_output=True,
-        text=True,
-    )
-    if result.returncode != 0:
-        return []
-    return [line for line in result.stdout.splitlines() if line]
-
-
 def kill_session(name: str) -> None:
     tmux = get_settings().tmux_bin
     subprocess.run([tmux, "kill-session", "-t", name], check=True)

src/handler/control/worker.py

@@ -1,11 +1,13 @@
-"""The control-container worker: executes commands the API enqueues.
+"""The control/worker container: executes commands the API enqueues and supervises
+headless claude runs.
 
-The API (in its own container) has no ``git``/``tmux``/``claude`` and does not own the
+The API (in its own container) has no ``git``/``claude``, so it cannot run control
-tmux sessions, so it cannot run control actions directly. Instead it writes a ``queued``
+actions directly. Instead it writes a ``queued`` row to the ``commands`` table; any
-row to the ``commands`` table; this worker — running in the control container — claims each
+worker claims each row (multi-worker safe — ``FOR UPDATE SKIP LOCKED`` + slot-aware
-row, dispatches it to the *same* control functions the CLI uses (``spawn``/``poller``/
+claim filters), dispatches it to the *same* control functions the CLI uses (``spawn``/
-``skills_gen``/``repo.record_approval``), and writes the result or error back. It also runs
+``poller``/``skills_gen``/``repo.record_approval``), and writes the result or error
-the periodic CI sweep, subsuming the old ``poll-ci --watch`` loop.
+back. It also heartbeats + reaps dead workers' runs, syncs claude credentials, and runs
+the periodic CI sweep.
 
 Every command runs in isolation: one bad command is recorded as ``failed`` and never stops
 the loop. ``execute_command`` is the pure dispatch seam (given a claimed command dict,
@@ -15,12 +17,24 @@ returns a JSON-safe result or raises); ``drain``/``run`` are the claim+finish pl
 from __future__ import annotations
 
 import os
+import secrets
+import socket
 import time
 from datetime import UTC, datetime, timedelta
 
+from ..config import get_settings
 from ..db import repository as repo
 from ..db.engine import connection
-from . import gitops, login, poller, reposync, skills_gen, spawn, tmux
+from . import credsync, gitops, login, poller, reposync, skills_gen, spawn
+
+# Command types that launch a claude run and therefore need a free slot on this worker.
+# A worker with all slots busy leaves these queued for a less-loaded worker to claim.
+_RUN_COMMANDS = ("spawn", "resume", "mise_init")
+
+
+def make_worker_id() -> str:
+    """A stable-for-this-process, unique-across-containers worker id."""
+    return f"worker-{socket.gethostname()}-{os.getpid()}-{secrets.token_hex(2)}"
 
 
 class CommandError(Exception):
@@ -43,6 +57,7 @@ def _cmd_spawn(command: dict) -> dict:
         worktree_branch=p.get("worktree"),
         task=p.get("task"),
         role=p.get("role"),
+        worker_id=command.get("claimed_by"),
     )
     result = {
         "agent_id": agent["id"],
@@ -73,11 +88,15 @@ def _cmd_resume(command: dict) -> dict:
         agent = repo.get_agent_by_name(conn, command["project_id"], name)
     if agent is None:
         raise CommandError(f"agent '{name}' not found in project '{command['project_id']}'")
-    ok, detail = spawn.resume(agent, answer)
+    ok, detail = spawn.resume(agent, answer, worker_id=command.get("claimed_by"))
-    if ok:
+    if not ok:
-        with connection() as conn:
+        # A resume that couldn't be delivered is a FAILED command, not a quiet no-op —
-            repo.set_agent_status(conn, agent["id"], "working")
+        # the operator's input must never vanish silently (the old tmux path typed into
-    return {"resumed": ok, "detail": detail}
+        # dead panes and reported success).
+        raise CommandError(detail)
+    with connection() as conn:
+        repo.set_agent_status(conn, agent["id"], "working")
+    return {"resumed": True, "detail": detail}
 
 
 def _record_verdict(command: dict, status: str) -> dict:
@@ -219,6 +238,13 @@ def _cmd_login_submit(command: dict) -> dict:
         # Surface the pane tail so the operator can see why claude rejected the code.
         detail = result.get("output") or "claude did not confirm a successful login"
         raise CommandError(f"login not confirmed — {detail}")
+    # Publish the fresh credentials so every other worker container can materialize
+    # them (multi-worker: the login ran here, but any worker may run the next agent).
+    try:
+        result["credentials_published"] = credsync.upload()
+    except Exception as exc:  # noqa: BLE001 - login succeeded; publishing is best-effort
+        result["credentials_published"] = False
+        result["credsync_note"] = str(exc)
     return result
 
 
@@ -274,6 +300,63 @@ def _run_one(command: dict) -> None:
             repo.finish_command(conn, command["id"], "failed", error=str(exc))
 
 
+def heartbeat(worker_id: str) -> None:
+    """Refresh this worker's registry row — the reaper's proof-of-life."""
+    with connection() as conn:
+        active = len(repo.list_running_runs(conn, worker_id=worker_id))
+        repo.upsert_worker_heartbeat(
+            conn,
+            worker_id,
+            hostname=socket.gethostname(),
+            pid=os.getpid(),
+            max_runs=get_settings().max_concurrent_runs,
+            active_runs=active,
+        )
+
+
+def reap_dead_workers(now: datetime | None = None) -> int:
+    """Settle the record for workers that stopped heartbeating: their ``running`` runs
+    become ``crashed``, and agents those runs left in ``working`` become ``crashed`` too.
+
+    Any surviving worker may reap any dead one — ``finish_run``'s still-running guard
+    makes concurrent reapers idempotent. Deliberately NO auto-requeue: a half-done run
+    may have already pushed commits, so re-running it isn't safe to assume; the operator
+    sees the crashed badge (with the frozen last_output as evidence) and resumes
+    explicitly. Agents in ``paused_for_input``/``blocked`` keep their status — those are
+    still accurate descriptions of what the agent needs. Returns runs reaped.
+    """
+    now = now or datetime.now(UTC)
+    cutoff = now - timedelta(seconds=get_settings().worker_stale_after)
+    reaped = 0
+    with connection() as conn:
+        stale = repo.list_stale_workers(conn, cutoff)
+    for dead in stale:
+        with connection() as conn:
+            for run in repo.list_running_runs(conn, worker_id=dead["id"]):
+                if not repo.finish_run(conn, run["id"], "crashed"):
+                    continue  # another reaper won the race
+                agent = repo.get_agent_by_id(conn, run["agent_id"])
+                if agent is not None and agent["status"] == "working":
+                    repo.set_agent_status(conn, agent["id"], "crashed")
+                repo.insert_agent_event(
+                    conn,
+                    run["agent_id"],
+                    run["id"],
+                    seq=0,  # runner-generated, outside the supervisor's line counter
+                    type="worker",
+                    payload={
+                        "notice": "worker died mid-run; run marked crashed",
+                        "worker_id": dead["id"],
+                    },
+                    session_id=run["session_id"],
+                )
+                reaped += 1
+            # Row served its purpose; dropping it keeps the registry to live workers
+            # (and stops every future pass re-scanning long-dead ids).
+            repo.delete_worker(conn, dead["id"])
+    return reaped
+
+
 def fire_due_schedules(now: datetime | None = None) -> int:
     """Enqueue a spawn command for every schedule whose ``next_run_at`` has passed.
 
@@ -313,43 +396,6 @@ def fire_due_schedules(now: datetime | None = None) -> int:
     return fired
 
 
-# How many trailing pane lines to snapshot — enough to show the current screen (a menu, a
-# prompt, the tail of the last command) without bloating the row.
-_PANE_TAIL_LINES = 40
-
-
-def _pane_tail(pane: str, lines: int = _PANE_TAIL_LINES) -> str:
-    """The last ``lines`` of a captured pane, trailing blank lines trimmed so an idle
-    screen doesn't store as a wall of whitespace."""
-    rows = (pane or "").splitlines()
-    while rows and not rows[-1].strip():
-        rows.pop()
-    return "\n".join(rows[-lines:])
-
-
-def capture_agent_output() -> int:
-    """Snapshot each working agent's live tmux pane tail into the DB.
-
-    The tmux socket lives only in the control container, so this is the one channel the
-    API/UI have onto what a running — or wedged — agent is actually doing: an agent stuck
-    on claude's first-run theme picker surfaces as that screen instead of a misleading
-    green 'working'. A missing session is skipped (its process is gone). Returns the count
-    updated.
-    """
-    with connection() as conn:
-        working = repo.list_agents_by_status(conn, "working")
-    updated = 0
-    for agent in working:
-        session = tmux.session_name(agent["project_id"], agent["name"])
-        if not tmux.has_session(session):
-            continue
-        tail = _pane_tail(tmux.capture_pane(session))
-        with connection() as conn:
-            repo.update_agent_output(conn, agent["id"], tail)
-        updated += 1
-    return updated
-
-
 def drain(worker_id: str, limit: int | None = None) -> int:
     """Claim and run queued commands until the queue is empty (or ``limit`` reached).
 
@@ -360,7 +406,9 @@ def drain(worker_id: str, limit: int | None = None) -> int:
     processed = 0
     while limit is None or processed < limit:
         with connection() as conn:
-            command = repo.claim_next_command(conn, worker_id)
+            command = repo.claim_next_command(
+                conn, worker_id, types_excluded=_full_slot_exclusions(worker_id)
+            )
         if command is None:
             break
         _run_one(command)
@@ -368,36 +416,66 @@ def drain(worker_id: str, limit: int | None = None) -> int:
     return processed
 
 
+def _full_slot_exclusions(worker_id: str) -> tuple[str, ...]:
+    """Command types this worker must not claim right now.
+
+    Runs are supervised in-process, so a worker at ``max_concurrent_runs`` skips
+    claiming run-starting commands — they stay queued for a worker with a free slot. Slot
+    accounting is DB-driven (this worker's ``running`` runs), so it needs no in-memory
+    registry and survives restarts (a fresh process gets a fresh worker id; stale rows
+    belong to the old id and are the reaper's problem).
+    """
+    with connection() as conn:
+        active = len(repo.list_running_runs(conn, worker_id=worker_id))
+    if active >= get_settings().max_concurrent_runs:
+        return _RUN_COMMANDS
+    return ()
+
+
 def run(
     worker_id: str | None = None,
     poll_interval: float = 2.0,
     ci_interval: float = 30.0,
-    capture_interval: float = 2.0,
+    credsync_interval: float = 30.0,
+    reap_interval: float = 15.0,
     iterations: int | None = None,
 ) -> None:
-    """The control-container main loop: drain the command queue, snapshot live agent
+    """The control-container main loop: drain the command queue, sync claude
-    output, and sweep CI periodically.
+    credentials, heartbeat + reap dead workers, and sweep CI.
 
     ``iterations`` bounds the loop for tests; production runs unbounded. Sleeps
     ``poll_interval`` only when a pass found no commands, so bursts drain promptly.
     """
-    worker_id = worker_id or f"worker-{os.getpid()}"
+    worker_id = worker_id or make_worker_id()
     last_ci = 0.0
-    last_capture = 0.0
+    last_credsync = 0.0
+    last_reap = 0.0
     count = 0
     while iterations is None or count < iterations:
+        try:
+            heartbeat(worker_id)
+        except Exception:  # noqa: BLE001 - a heartbeat hiccup must not kill the worker
+            pass
+        if reap_interval > 0 and time.monotonic() - last_reap >= reap_interval:
+            try:
+                reap_dead_workers()
+            except Exception:  # noqa: BLE001 - a reap hiccup must not kill the worker
+                pass
+            last_reap = time.monotonic()
         try:
             fire_due_schedules()
         except Exception:  # noqa: BLE001 - a schedule hiccup must not kill the worker
             pass
         did_work = drain(worker_id) > 0
         now = time.monotonic()
-        if capture_interval > 0 and now - last_capture >= capture_interval:
+        if credsync_interval > 0 and (last_credsync == 0.0 or now - last_credsync >= credsync_interval):
+            # First pass runs immediately: a fresh worker container must materialize the
+            # claude credentials before it claims its first spawn.
             try:
-                capture_agent_output()
+                credsync.refresh()
-            except Exception:  # noqa: BLE001 - a capture hiccup must not kill the worker
+            except Exception:  # noqa: BLE001 - cred sync must not kill the worker
                 pass
-            last_capture = now
+            last_credsync = now
         if ci_interval > 0 and now - last_ci >= ci_interval:
             try:
                 poller.sweep()

src/handler/db/repository.py

@@ -22,6 +22,8 @@ from typing import Any
 from sqlalchemy import Connection, select
 
 from .tables import (
+    agent_events,
+    agent_runs,
     agents,
     approvals,
     checkmarks,
@@ -29,8 +31,11 @@ from .tables import (
     forge_hosts,
     log_entries,
     projects,
+    runtime_secrets,
     schedules,
+    session_archives,
     shared_context,
+    workers,
 )
 from .upsert import upsert_checkmark
 
@@ -373,6 +378,10 @@ def _purge_agent_dependents(conn: Connection, agent_ids: list[int]) -> None:
     conn.execute(approvals.delete().where(approvals.c.approved_by_agent_id.in_(agent_ids)))
     conn.execute(checkmarks.delete().where(checkmarks.c.agent_id.in_(agent_ids)))
     conn.execute(log_entries.delete().where(log_entries.c.agent_id.in_(agent_ids)))
+    # Headless-runner rows: events reference runs, so they go first.
+    conn.execute(agent_events.delete().where(agent_events.c.agent_id.in_(agent_ids)))
+    conn.execute(agent_runs.delete().where(agent_runs.c.agent_id.in_(agent_ids)))
+    conn.execute(session_archives.delete().where(session_archives.c.agent_id.in_(agent_ids)))
 
 
 def delete_project(conn: Connection, project_id: str) -> bool:
@@ -424,8 +433,13 @@ def enqueue_command(
     agent_name: str | None = None,
     payload: dict | None = None,
     requested_by: str | None = None,
+    target_worker: str | None = None,
 ) -> dict:
-    """Insert a ``queued`` control command for the worker to pick up. Returns the row."""
+    """Insert a ``queued`` control command for the worker to pick up. Returns the row.
+
+    ``target_worker`` pins the command to one worker id (used by login_submit, which must
+    reach the container holding the live login tmux session); null = any worker.
+    """
     result = conn.execute(
         commands.insert().values(
             project_id=project_id,
@@ -434,6 +448,7 @@ def enqueue_command(
             payload=payload,
             status="queued",
             requested_by=requested_by,
+            target_worker=target_worker,
             created_at=_now(),
         )
     )
@@ -457,20 +472,34 @@ def list_commands(
     return [dict(r._mapping) for r in rows]
 
 
-def claim_next_command(conn: Connection, worker_id: str) -> dict | None:
+def claim_next_command(
+    conn: Connection,
+    worker_id: str,
+    types_excluded: tuple[str, ...] = (),
+) -> dict | None:
     """Atomically claim the oldest queued command, flipping it to ``running``.
 
     Postgres uses ``FOR UPDATE SKIP LOCKED`` so multiple workers never grab the same row;
     on SQLite (single writer per transaction) the guarded ``WHERE status='queued'`` update
     plus a rowcount check is enough. Returns the claimed row, or ``None`` when the queue is
     empty or another worker won the race.
+
+    A command whose ``target_worker`` is set is only claimable by that worker (the login
+    flow's two steps must land on the same container). ``types_excluded`` lets a worker
+    with no free run slots skip claiming commands that would launch a new run, leaving
+    them for a less-loaded worker.
     """
     sel = (
         select(commands.c.id)
-        .where(commands.c.status == "queued")
+        .where(
+            commands.c.status == "queued",
+            (commands.c.target_worker.is_(None)) | (commands.c.target_worker == worker_id),
+        )
         .order_by(commands.c.id.asc())
         .limit(1)
     )
+    if types_excluded:
+        sel = sel.where(commands.c.type.not_in(types_excluded))
     if conn.dialect.name == "postgresql":
         sel = sel.with_for_update(skip_locked=True)
     row = conn.execute(sel).first()
@@ -659,3 +688,227 @@ def mark_schedule_run(
             last_run_at=last_run_at, next_run_at=next_run_at, last_command_id=last_command_id
         )
     )
+
+
+# ------------------------------------------------- headless runner (workers/runs/events)
+
+
+def upsert_worker_heartbeat(
+    conn: Connection,
+    worker_id: str,
+    *,
+    hostname: str | None = None,
+    pid: int | None = None,
+    max_runs: int | None = None,
+    active_runs: int | None = None,
+) -> None:
+    """Register a worker or refresh its heartbeat — the reaper's liveness signal."""
+    now = _now()
+    result = conn.execute(
+        workers.update()
+        .where(workers.c.id == worker_id)
+        .values(
+            hostname=hostname, pid=pid, max_runs=max_runs,
+            active_runs=active_runs, heartbeat_at=now,
+        )
+    )
+    if result.rowcount == 0:
+        conn.execute(
+            workers.insert().values(
+                id=worker_id, hostname=hostname, pid=pid, max_runs=max_runs,
+                active_runs=active_runs, started_at=now, heartbeat_at=now,
+            )
+        )
+
+
+def list_stale_workers(conn: Connection, cutoff: datetime) -> list[dict]:
+    """Workers whose heartbeat predates ``cutoff`` — reaper input."""
+    rows = conn.execute(select(workers).where(workers.c.heartbeat_at < cutoff)).all()
+    return [dict(r._mapping) for r in rows]
+
+
+def delete_worker(conn: Connection, worker_id: str) -> bool:
+    """Drop a (dead) worker's registry row after its runs have been settled."""
+    result = conn.execute(workers.delete().where(workers.c.id == worker_id))
+    return result.rowcount > 0
+
+
+def create_run(
+    conn: Connection, agent_id: int, session_id: str, worker_id: str, kind: str
+) -> dict:
+    """Open an ``agent_runs`` row for a launching headless invocation."""
+    result = conn.execute(
+        agent_runs.insert().values(
+            agent_id=agent_id,
+            session_id=session_id,
+            worker_id=worker_id,
+            kind=kind,
+            status="running",
+            cancel_requested=False,
+            started_at=_now(),
+        )
+    )
+    return get_run(conn, result.inserted_primary_key[0])
+
+
+def get_run(conn: Connection, run_id: int) -> dict | None:
+    row = conn.execute(select(agent_runs).where(agent_runs.c.id == run_id)).first()
+    return _row_to_dict(row)
+
+
+def finish_run(
+    conn: Connection,
+    run_id: int,
+    status: str,
+    exit_code: int | None = None,
+    result: dict | None = None,
+) -> bool:
+    """Close a run — only if still ``running``, so a reaper's ``crashed`` verdict and the
+    supervisor's own exit reconciliation can race without the loser clobbering the winner."""
+    res = conn.execute(
+        agent_runs.update()
+        .where(agent_runs.c.id == run_id, agent_runs.c.status == "running")
+        .values(status=status, exit_code=exit_code, result=result, finished_at=_now())
+    )
+    return res.rowcount > 0
+
+
+def request_run_cancel(conn: Connection, run_id: int) -> bool:
+    """Flag a running run for cancellation; its owning supervisor polls and SIGTERMs."""
+    result = conn.execute(
+        agent_runs.update()
+        .where(agent_runs.c.id == run_id, agent_runs.c.status == "running")
+        .values(cancel_requested=True)
+    )
+    return result.rowcount > 0
+
+
+def get_cancel_requested(conn: Connection, run_id: int) -> bool:
+    row = conn.execute(
+        select(agent_runs.c.cancel_requested).where(agent_runs.c.id == run_id)
+    ).first()
+    return bool(row[0]) if row is not None else False
+
+
+def list_running_runs(conn: Connection, worker_id: str | None = None) -> list[dict]:
+    """Runs currently ``running`` — all of them, or one worker's (the reaper's sweep)."""
+    stmt = select(agent_runs).where(agent_runs.c.status == "running")
+    if worker_id is not None:
+        stmt = stmt.where(agent_runs.c.worker_id == worker_id)
+    rows = conn.execute(stmt.order_by(agent_runs.c.id)).all()
+    return [dict(r._mapping) for r in rows]
+
+
+def get_latest_run(conn: Connection, agent_id: int) -> dict | None:
+    row = conn.execute(
+        select(agent_runs)
+        .where(agent_runs.c.agent_id == agent_id)
+        .order_by(agent_runs.c.id.desc())
+        .limit(1)
+    ).first()
+    return _row_to_dict(row)
+
+
+def set_agent_session(
+    conn: Connection, agent_id: int, session_id: str, worker_id: str
+) -> None:
+    """Record the claude session UUID + supervising worker on the agent row."""
+    conn.execute(
+        agents.update()
+        .where(agents.c.id == agent_id)
+        .values(session_id=session_id, worker_id=worker_id)
+    )
+
+
+def insert_agent_event(
+    conn: Connection,
+    agent_id: int,
+    run_id: int,
+    seq: int,
+    type: str,
+    payload: dict | None = None,
+    session_id: str | None = None,
+) -> int:
+    result = conn.execute(
+        agent_events.insert().values(
+            agent_id=agent_id,
+            run_id=run_id,
+            session_id=session_id,
+            seq=seq,
+            type=type,
+            payload=payload,
+            created_at=_now(),
+        )
+    )
+    return result.inserted_primary_key[0]
+
+
+def list_agent_events(
+    conn: Connection, agent_id: int, after_id: int = 0, limit: int = 200
+) -> list[dict]:
+    """Events for an agent in insertion order, cursor-paged by row id (the UI polls with
+    ``after_id`` = the last id it has, so each poll returns only what's new)."""
+    rows = conn.execute(
+        select(agent_events)
+        .where(agent_events.c.agent_id == agent_id, agent_events.c.id > after_id)
+        .order_by(agent_events.c.id.asc())
+        .limit(limit)
+    ).all()
+    return [dict(r._mapping) for r in rows]
+
+
+def upsert_session_archive(
+    conn: Connection, agent_id: int, session_id: str, archive: bytes
+) -> None:
+    """Store (replace) the agent's latest claude session archive."""
+    now = _now()
+    result = conn.execute(
+        session_archives.update()
+        .where(session_archives.c.agent_id == agent_id)
+        .values(session_id=session_id, archive=archive, bytes=len(archive), updated_at=now)
+    )
+    if result.rowcount == 0:
+        conn.execute(
+            session_archives.insert().values(
+                agent_id=agent_id, session_id=session_id, archive=archive,
+                bytes=len(archive), updated_at=now,
+            )
+        )
+
+
+def get_session_archive(conn: Connection, agent_id: int) -> dict | None:
+    row = conn.execute(
+        select(session_archives).where(session_archives.c.agent_id == agent_id)
+    ).first()
+    return _row_to_dict(row)
+
+
+def upsert_runtime_secret(conn: Connection, key: str, value_enc: str) -> None:
+    """Store an encrypted control-plane secret (the caller encrypts via secretstore)."""
+    now = _now()
+    result = conn.execute(
+        runtime_secrets.update()
+        .where(runtime_secrets.c.key == key)
+        .values(value_enc=value_enc, updated_at=now)
+    )
+    if result.rowcount == 0:
+        conn.execute(
+            runtime_secrets.insert().values(key=key, value_enc=value_enc, updated_at=now)
+        )
+
+
+def get_runtime_secret(conn: Connection, key: str) -> dict | None:
+    row = conn.execute(select(runtime_secrets).where(runtime_secrets.c.key == key)).first()
+    return _row_to_dict(row)
+
+
+def get_latest_finished_command(conn: Connection, type: str) -> dict | None:
+    """The most recent ``done`` command of a type (login pinning reads login_start's
+    ``claimed_by`` to route login_submit to the same worker container)."""
+    row = conn.execute(
+        select(commands)
+        .where(commands.c.type == type, commands.c.status == "done")
+        .order_by(commands.c.id.desc())
+        .limit(1)
+    ).first()
+    return _row_to_dict(row)

src/handler/db/tables.py

@@ -14,6 +14,7 @@ from sqlalchemy import (
     Column,
     ForeignKey,
     Index,
+    LargeBinary,
     MetaData,
     String,
     Table,
@@ -27,7 +28,9 @@ metadata = MetaData()
 
 # Status vocabularies kept as free TEXT (README uses plain strings, not PG enums, so
 # both dialects match). CheckConstraints make the allowed sets explicit and portable.
-AGENT_STATUSES = ("working", "paused_for_input", "blocked", "done")
+# ``crashed`` is reserved for the reaper: it marks an agent whose owning worker went
+# silent mid-run — never a normal exit, which reconciles to done/blocked instead.
+AGENT_STATUSES = ("working", "paused_for_input", "blocked", "done", "crashed")
 GATE_STATUSES = ("pass", "fail", "unknown")
 CI_STATUSES = ("not_applicable", "pending", "pass", "fail")
 VISIBILITIES = ("project", "global")
@@ -55,6 +58,10 @@ COMMAND_TYPES = (
 COMMAND_STATUSES = ("queued", "running", "done", "failed")
 # Forge families a host can belong to (drives per-host token env conventions).
 FORGE_TYPES = ("github", "gitlab", "gitea", "forgejo", "bitbucket")
+# One agent_runs row per headless ``claude -p`` invocation (spawn or resume).
+# ``canceled`` = an operator kill; ``crashed`` = the reaper found the owning worker dead.
+RUN_KINDS = ("spawn", "resume")
+RUN_STATUSES = ("running", "completed", "failed", "crashed", "canceled")
 
 
 def _in(column: str, values: tuple[str, ...]) -> str:
@@ -87,8 +94,14 @@ agents = Table(
     # A periodic snapshot of the agent's live tmux pane tail (last ~40 lines), refreshed by
     # the control worker's poll loop. The tmux socket lives only in the control container,
     # so this DB column is how the API/UI see what a running — or wedged — agent is doing.
+    # Headless runs reuse it, derived from the latest assistant text instead of a pane tail.
     Column("last_output", String),
     Column("output_at", PortableTimestamp),
+    # Headless runner (null on legacy tmux agents — the rollout discriminator): the claude
+    # session UUID pre-assigned at spawn (stable across resumes), and the worker currently
+    # (or last) supervising a run for this agent.
+    Column("session_id", String),
+    Column("worker_id", String),
     Column("created_at", PortableTimestamp, nullable=False, server_default=func.now()),
     UniqueConstraint("project_id", "name", name="uq_agents_project_name"),
     CheckConstraint(_in("status", AGENT_STATUSES), name="ck_agents_status"),
@@ -192,6 +205,9 @@ commands = Table(
     Column("error", String),
     Column("requested_by", String),  # actor label, e.g. "operator:web"
     Column("claimed_by", String),  # worker id that claimed the command
+    # Pins a command to one worker (null = any). login_submit must run on the worker that
+    # ran login_start — the live tmux login session exists only in that container.
+    Column("target_worker", String),
     Column("created_at", PortableTimestamp, nullable=False, server_default=func.now()),
     Column("claimed_at", PortableTimestamp),
     Column("finished_at", PortableTimestamp),
@@ -227,6 +243,91 @@ forge_hosts = Table(
     CheckConstraint(_in("forge_type", FORGE_TYPES), name="ck_forge_hosts_type"),
 )
 
+# Control-plane key/value secrets, Fernet-encrypted like forge_hosts tokens (see
+# ``handler.secretstore``). Holds the claude OAuth credential bundle so every worker
+# container can materialize it locally — the login flow runs on ONE worker, but all of
+# them need to run ``claude``. Never exposed by any API route.
+runtime_secrets = Table(
+    "runtime_secrets",
+    metadata,
+    Column("key", String, primary_key=True),
+    Column("value_enc", String, nullable=False),
+    Column("updated_at", PortableTimestamp, nullable=False, server_default=func.now()),
+)
+
+# Worker registry + heartbeat (headless runner). Each worker container upserts its row
+# every loop pass; the reaper marks a worker's running runs (and their agents) ``crashed``
+# when ``heartbeat_at`` goes stale — the positive-liveness replacement for tmux scraping.
+workers = Table(
+    "workers",
+    metadata,
+    Column("id", String, primary_key=True),  # "worker-<host>-<pid>-<rand>"
+    Column("hostname", String),
+    Column("pid", BigInteger),
+    Column("max_runs", BigInteger),
+    Column("active_runs", BigInteger),
+    Column("started_at", PortableTimestamp, nullable=False, server_default=func.now()),
+    Column("heartbeat_at", PortableTimestamp, nullable=False, server_default=func.now()),
+)
+
+# One row per headless ``claude -p`` invocation. The spawn/resume *command* finishes at
+# launch (fire-and-forget, matching tmux semantics); the run row is what tracks the
+# process's actual life — status, exit code, and the final result event.
+agent_runs = Table(
+    "agent_runs",
+    metadata,
+    Column("id", PortableBigInt, primary_key=True, autoincrement=True),
+    Column("agent_id", BigInteger, ForeignKey("agents.id"), nullable=False),
+    Column("session_id", String, nullable=False),
+    Column("worker_id", String, nullable=False),
+    Column("kind", String, nullable=False),
+    Column("status", String, nullable=False, server_default="running"),
+    # Cross-worker kill: any worker/API sets this; the owning supervisor polls it and
+    # SIGTERMs its own child — nobody signals a process they don't own.
+    Column("cancel_requested", Boolean, nullable=False, server_default="0"),
+    Column("exit_code", BigInteger),
+    Column("result", PortableJSON),  # the stream's result event (cost/turns/is_error/text)
+    Column("started_at", PortableTimestamp, nullable=False, server_default=func.now()),
+    Column("finished_at", PortableTimestamp),
+    CheckConstraint(_in("kind", RUN_KINDS), name="ck_agent_runs_kind"),
+    CheckConstraint(_in("status", RUN_STATUSES), name="ck_agent_runs_status"),
+    Index("ix_agent_runs_status_worker", "status", "worker_id"),
+)
+
+# The persisted event stream — one row per stream-json stdout line of a run, in order.
+# This is what the UI's log/event panel reads; ``type`` mirrors the stream's top-level
+# type (system/assistant/user/result), plus ``hook`` (--include-hook-events), ``worker``
+# (runner-generated notices: crash marks, archive failures, fallback resumes) and ``raw``
+# (an unparseable line, stored verbatim — the parser never drops data).
+agent_events = Table(
+    "agent_events",
+    metadata,
+    Column("id", PortableBigInt, primary_key=True, autoincrement=True),
+    Column("agent_id", BigInteger, ForeignKey("agents.id"), nullable=False),
+    Column("run_id", BigInteger, ForeignKey("agent_runs.id"), nullable=False),
+    Column("session_id", String),
+    Column("seq", BigInteger, nullable=False),  # per-run stdout line counter
+    Column("type", String, nullable=False),
+    Column("payload", PortableJSON),
+    Column("created_at", PortableTimestamp, nullable=False, server_default=func.now()),
+    Index("ix_agent_events_agent_id", "agent_id", "id"),
+)
+
+# Latest claude session archive per agent — the tar.gz of ``<sid>.jsonl`` + its sidecar
+# dir from ``~/.claude/projects/<munged-cwd>/``. Uploaded by the supervising worker
+# (periodically and at run end) and materialized by whichever worker claims the next
+# resume, so ``--resume`` works cross-worker with no shared filesystem (README: DB is the
+# single source of truth; observed sizes are KBs-to-low-MBs).
+session_archives = Table(
+    "session_archives",
+    metadata,
+    Column("agent_id", BigInteger, ForeignKey("agents.id"), primary_key=True),
+    Column("session_id", String, nullable=False),
+    Column("archive", LargeBinary, nullable=False),
+    Column("bytes", BigInteger, nullable=False),
+    Column("updated_at", PortableTimestamp, nullable=False, server_default=func.now()),
+)
+
 # Recurring agent spawns. The worker checks for due rows on every loop pass and enqueues
 # an ordinary ``spawn`` command per firing (so scheduled runs show up in the Activity
 # audit trail like any other control action). Agent names must be unique per project, so

src/handler/migrations/versions/0008_headless_runs.py

@@ -0,0 +1,128 @@
+"""headless runner: workers, runs, events, session archives
+
+Revision ID: 0008_headless_runs
+Revises: 0007_agent_pane_output
+Create Date: 2026-07-21
+
+Dormant schema for the headless ``claude -p --output-format stream-json`` runner
+(nothing writes these until the runner is enabled):
+
+- ``workers`` — worker registry + heartbeat, the reaper's liveness input.
+- ``agent_runs`` — one row per headless invocation (spawn or resume), tracking the
+  process's real life: status, exit code, cancel flag, final result event.
+- ``agent_events`` — the persisted stream-json event log the UI reads.
+- ``session_archives`` — latest claude session tar.gz per agent, so ``--resume`` works
+  from any worker with no shared filesystem.
+- ``agents`` gains ``session_id`` (null = legacy tmux agent, the rollout discriminator)
+  and ``worker_id``; ``commands`` gains ``target_worker`` (pins login_submit to the
+  worker holding the live login session).
+- ``crashed`` joins the agent status vocabulary (reaper-set); the CHECK constraint
+  swaps go through ``batch_alter_table`` so SQLite recreates while Postgres alters in
+  place (same pattern as 0006's command type).
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+from handler.db.types import PortableBigInt, PortableJSON, PortableTimestamp
+
+revision: str = "0008_headless_runs"
+down_revision: str | None = "0007_agent_pane_output"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+OLD_AGENT_STATUSES = "'working', 'paused_for_input', 'blocked', 'done'"
+NEW_AGENT_STATUSES = "'working', 'paused_for_input', 'blocked', 'done', 'crashed'"
+RUN_KINDS = "'spawn', 'resume'"
+RUN_STATUSES = "'running', 'completed', 'failed', 'crashed', 'canceled'"
+
+
+def upgrade() -> None:
+    op.create_table(
+        "workers",
+        sa.Column("id", sa.String(), primary_key=True),
+        sa.Column("hostname", sa.String()),
+        sa.Column("pid", sa.BigInteger()),
+        sa.Column("max_runs", sa.BigInteger()),
+        sa.Column("active_runs", sa.BigInteger()),
+        sa.Column("started_at", PortableTimestamp, nullable=False, server_default=sa.func.now()),
+        sa.Column("heartbeat_at", PortableTimestamp, nullable=False, server_default=sa.func.now()),
+    )
+    op.create_table(
+        "agent_runs",
+        sa.Column("id", PortableBigInt, primary_key=True, autoincrement=True),
+        sa.Column("agent_id", sa.BigInteger(), sa.ForeignKey("agents.id"), nullable=False),
+        sa.Column("session_id", sa.String(), nullable=False),
+        sa.Column("worker_id", sa.String(), nullable=False),
+        sa.Column("kind", sa.String(), nullable=False),
+        sa.Column("status", sa.String(), nullable=False, server_default="running"),
+        sa.Column("cancel_requested", sa.Boolean(), nullable=False, server_default="0"),
+        sa.Column("exit_code", sa.BigInteger()),
+        sa.Column("result", PortableJSON),
+        sa.Column("started_at", PortableTimestamp, nullable=False, server_default=sa.func.now()),
+        sa.Column("finished_at", PortableTimestamp),
+        sa.CheckConstraint(f"kind IN ({RUN_KINDS})", name="ck_agent_runs_kind"),
+        sa.CheckConstraint(f"status IN ({RUN_STATUSES})", name="ck_agent_runs_status"),
+    )
+    op.create_index(
+        "ix_agent_runs_status_worker", "agent_runs", ["status", "worker_id"]
+    )
+    op.create_table(
+        "agent_events",
+        sa.Column("id", PortableBigInt, primary_key=True, autoincrement=True),
+        sa.Column("agent_id", sa.BigInteger(), sa.ForeignKey("agents.id"), nullable=False),
+        sa.Column("run_id", sa.BigInteger(), sa.ForeignKey("agent_runs.id"), nullable=False),
+        sa.Column("session_id", sa.String()),
+        sa.Column("seq", sa.BigInteger(), nullable=False),
+        sa.Column("type", sa.String(), nullable=False),
+        sa.Column("payload", PortableJSON),
+        sa.Column("created_at", PortableTimestamp, nullable=False, server_default=sa.func.now()),
+    )
+    op.create_index("ix_agent_events_agent_id", "agent_events", ["agent_id", "id"])
+    op.create_table(
+        "session_archives",
+        sa.Column("agent_id", sa.BigInteger(), sa.ForeignKey("agents.id"), primary_key=True),
+        sa.Column("session_id", sa.String(), nullable=False),
+        sa.Column("archive", sa.LargeBinary(), nullable=False),
+        sa.Column("bytes", sa.BigInteger(), nullable=False),
+        sa.Column("updated_at", PortableTimestamp, nullable=False, server_default=sa.func.now()),
+    )
+    op.add_column("agents", sa.Column("session_id", sa.String()))
+    op.add_column("agents", sa.Column("worker_id", sa.String()))
+    op.add_column("commands", sa.Column("target_worker", sa.String()))
+    with op.batch_alter_table("agents", schema=None) as batch_op:
+        batch_op.drop_constraint("ck_agents_status", type_="check")
+        batch_op.create_check_constraint(
+            "ck_agents_status", f"status IN ({NEW_AGENT_STATUSES})"
+        )
+    with op.batch_alter_table("checkmarks", schema=None) as batch_op:
+        batch_op.drop_constraint("ck_checkmarks_status", type_="check")
+        batch_op.create_check_constraint(
+            "ck_checkmarks_status", f"status IN ({NEW_AGENT_STATUSES})"
+        )
+
+
+def downgrade() -> None:
+    with op.batch_alter_table("checkmarks", schema=None) as batch_op:
+        batch_op.drop_constraint("ck_checkmarks_status", type_="check")
+        batch_op.create_check_constraint(
+            "ck_checkmarks_status", f"status IN ({OLD_AGENT_STATUSES})"
+        )
+    with op.batch_alter_table("agents", schema=None) as batch_op:
+        batch_op.drop_constraint("ck_agents_status", type_="check")
+        batch_op.create_check_constraint(
+            "ck_agents_status", f"status IN ({OLD_AGENT_STATUSES})"
+        )
+    op.drop_column("commands", "target_worker")
+    op.drop_column("agents", "worker_id")
+    op.drop_column("agents", "session_id")
+    op.drop_table("session_archives")
+    op.drop_index("ix_agent_events_agent_id", table_name="agent_events")
+    op.drop_table("agent_events")
+    op.drop_index("ix_agent_runs_status_worker", table_name="agent_runs")
+    op.drop_table("agent_runs")
+    op.drop_table("workers")

src/handler/migrations/versions/0009_runtime_secrets.py

@@ -0,0 +1,39 @@
+"""runtime_secrets: encrypted control-plane key/value store
+
+Revision ID: 0009_runtime_secrets
+Revises: 0008_headless_runs
+Create Date: 2026-07-21
+
+A small Fernet-encrypted key/value table (values encrypted with ``HANDLER_SECRET_KEY``
+before insert, same policy as ``forge_hosts.token_enc``). First use: the claude OAuth
+credential bundle — the interactive ``/login`` flow completes on one worker container,
+and every other worker materializes the bundle from here at startup/refresh so any of
+them can run headless ``claude -p`` without shared files.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+from handler.db.types import PortableTimestamp
+
+revision: str = "0009_runtime_secrets"
+down_revision: str | None = "0008_headless_runs"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "runtime_secrets",
+        sa.Column("key", sa.String(), primary_key=True),
+        sa.Column("value_enc", sa.String(), nullable=False),
+        sa.Column("updated_at", PortableTimestamp, nullable=False, server_default=sa.func.now()),
+    )
+
+
+def downgrade() -> None:
+    op.drop_table("runtime_secrets")

tests/conftest.py

@@ -1,7 +1,8 @@
 """Shared fixtures. Everything runs on a fresh SQLite file per test, materialized via
 a *real* ``alembic upgrade head`` — so the migration path itself is under test, not
-just ``create_all``. No live claude/tmux/mise is ever touched: the three seams
+just ``create_all``. No live claude/tmux/mise is ever touched: the seams
-(``control.tmux``, ``hooks.verify``, ``control.spawn.resume``) are faked.
+(``control.headless.launch`` for runs, ``control.tmux`` for the login flow,
+``hooks.verify``, ``control.spawn.resume``) are faked.
 """
 
 from __future__ import annotations
@@ -111,20 +112,44 @@ def fake_tmux(monkeypatch):
     def send_enter(name):
         calls["send_enter"].append({"name": name})
 
-    def list_sessions():
-        return list(live)
-
     monkeypatch.setattr(tmux, "new_session", new_session)
     monkeypatch.setattr(tmux, "has_session", has_session)
     monkeypatch.setattr(tmux, "kill_session", kill_session)
     monkeypatch.setattr(tmux, "send_keys", send_keys)
     monkeypatch.setattr(tmux, "send_text", send_text)
     monkeypatch.setattr(tmux, "send_enter", send_enter)
-    monkeypatch.setattr(tmux, "list_sessions", list_sessions)
 
     return {"calls": calls, "live": live}
 
 
+@pytest.fixture
+def fake_launch(monkeypatch):
+    """Record ``headless.launch`` calls instead of spawning a claude subprocess.
+
+    Mirrors the real launch's DB side effects (run row + agent session/worker) so kill/
+    resume logic downstream of a fake spawn behaves like production, minus the process.
+    """
+    from handler.control import headless
+    from handler.db import repository as repo
+    from handler.db.engine import connection
+
+    calls: list[dict] = []
+
+    def launch(agent, *, kind, prompt, settings_path, env, worker_id, on_exit=None):
+        session_id = agent.get("session_id") if kind == "resume" else f"fake-sid-{len(calls) + 1}"
+        with connection() as conn:
+            run = repo.create_run(conn, agent["id"], session_id, worker_id, kind)
+            repo.set_agent_session(conn, agent["id"], session_id, worker_id)
+        calls.append(
+            {"agent": agent, "kind": kind, "prompt": prompt, "settings_path": settings_path,
+             "env": env, "worker_id": worker_id, "run": run}
+        )
+        return run
+
+    monkeypatch.setattr(headless, "launch", launch)
+    return calls
+
+
 @pytest.fixture
 def fake_gitops(monkeypatch):
     """Fake the git seam: record config/add/commit, return a controllable branch/sha."""

tests/fixtures/fake_claude.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""A stand-in ``claude`` binary for headless-runner tests.
+
+Wired in via the existing ``claude_bin`` setting (the same seam the tmux fakes used).
+Parses the real headless argv, emits a scripted ``--output-format stream-json`` stream on
+stdout, and writes a genuine session transcript + sidecar under
+``$HOME/.claude/projects/<munged-cwd>/`` so archive/materialize/resume paths exercise the
+real filesystem layout. Behavior is selected with ``FAKE_CLAUDE_MODE``:
+
+- ``success`` (default): init + assistant + result, exit 0.
+- ``error``: init + assistant + one garbage line, then exit 2 with no result event.
+- ``hang``: init, then sleep forever (the kill/cancel/reaper tests SIGTERM it).
+- ``slow``: like success with a pause between events (concurrency tests).
+- ``resume-fail``: a ``--resume`` invocation exits 1 before any assistant event
+  (exercises the context re-injection fallback).
+
+On ``--resume`` (outside resume-fail) the transcript materialized by the worker MUST
+already exist at the expected path — missing means cross-worker materialization broke,
+so the fake fails loudly, exit 3.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import signal
+import sys
+import time
+from pathlib import Path
+
+
+def _parse_argv(argv: list[str]) -> dict:
+    opts = {
+        "print": False,
+        "verbose": False,
+        "output_format": None,
+        "session_id": None,
+        "resume": None,
+        "settings": None,
+        "budget": None,
+        "prompt": None,
+    }
+    i = 0
+    while i < len(argv):
+        arg = argv[i]
+        if arg == "-p":
+            opts["print"] = True
+        elif arg == "--verbose":
+            opts["verbose"] = True
+        elif arg == "--output-format":
+            i += 1
+            opts["output_format"] = argv[i]
+        elif arg == "--session-id":
+            i += 1
+            opts["session_id"] = argv[i]
+        elif arg in ("--resume", "-r"):
+            i += 1
+            opts["resume"] = argv[i]
+        elif arg == "--settings":
+            i += 1
+            opts["settings"] = argv[i]
+        elif arg == "--max-budget-usd":
+            i += 1
+            opts["budget"] = argv[i]
+        elif arg == "--":
+            opts["prompt"] = " ".join(argv[i + 1 :])
+            break
+        else:
+            opts["prompt"] = arg
+        i += 1
+    return opts
+
+
+def _munged(cwd: str) -> str:
+    return cwd.replace("/", "-").replace(".", "-")
+
+
+def _emit(event: dict) -> None:
+    sys.stdout.write(json.dumps(event) + "\n")
+    sys.stdout.flush()
+
+
+def _write_transcript(session_id: str, prompt: str) -> None:
+    base = Path(os.path.expanduser("~")) / ".claude" / "projects" / _munged(os.getcwd())
+    base.mkdir(parents=True, exist_ok=True)
+    jsonl = base / f"{session_id}.jsonl"
+    with jsonl.open("a") as fh:
+        fh.write(json.dumps({"type": "user", "prompt": prompt}) + "\n")
+        fh.write(json.dumps({"type": "assistant", "text": f"handled: {prompt}"}) + "\n")
+    sidecar = base / session_id / "tool-results"
+    sidecar.mkdir(parents=True, exist_ok=True)
+    (sidecar / "result-1.txt").write_text("fake tool output\n")
+
+
+def main() -> int:
+    signal.signal(signal.SIGTERM, signal.SIG_DFL)
+    mode = os.environ.get("FAKE_CLAUDE_MODE", "success")
+    opts = _parse_argv(sys.argv[1:])
+    if not opts["print"] or opts["output_format"] != "stream-json":
+        sys.stderr.write("fake_claude: expected -p --output-format stream-json\n")
+        return 64
+
+    session_id = opts["session_id"] or opts["resume"] or "fake-session"
+    prompt = opts["prompt"] or ""
+
+    if mode == "resume-fail" and opts["resume"]:
+        sys.stderr.write("fake_claude: no conversation found to resume\n")
+        return 1
+
+    if opts["resume"]:
+        base = Path(os.path.expanduser("~")) / ".claude" / "projects" / _munged(os.getcwd())
+        if not (base / f"{session_id}.jsonl").exists():
+            sys.stderr.write(f"fake_claude: transcript missing at {base}\n")
+            return 3
+
+    _emit(
+        {
+            "type": "system",
+            "subtype": "init",
+            "session_id": session_id,
+            "cwd": os.getcwd(),
+            "tools": ["Bash", "Read", "Edit"],
+        }
+    )
+    if mode == "hang":
+        time.sleep(3600)
+        return 0
+    if mode == "slow":
+        time.sleep(float(os.environ.get("FAKE_CLAUDE_SLOW_SECONDS", "1.0")))
+
+    _emit(
+        {
+            "type": "assistant",
+            "session_id": session_id,
+            "message": {
+                "role": "assistant",
+                "content": [{"type": "text", "text": f"working on: {prompt}"}],
+            },
+        }
+    )
+    _write_transcript(session_id, prompt)
+
+    if mode == "error":
+        sys.stdout.write("this is not json\n")
+        sys.stdout.flush()
+        return 2
+
+    _emit(
+        {
+            "type": "result",
+            "subtype": "success",
+            "session_id": session_id,
+            "is_error": False,
+            "num_turns": 1,
+            "total_cost_usd": 0.01,
+            "result": f"done: {prompt}",
+        }
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tests/test_control_spawn.py

@@ -1,4 +1,7 @@
-"""Control-layer spawn: the hard test-task gate, settings generation, identity env."""
+"""Control-layer spawn: the hard test-task gate, settings generation, identity env.
+
+Spawns go through the ``fake_launch`` seam (conftest) — the headless analogue of the old
+fake tmux: it records the launch and mirrors its DB side effects, no subprocess."""
 
 from __future__ import annotations
 
@@ -24,36 +27,48 @@ def _write_mise(root, with_test=True):
     (root / ".mise.toml").write_text(body)
 
 
-def test_spawn_refuses_without_test_task(env, fake_tmux):
+def test_spawn_refuses_without_test_task(env, fake_launch):
     root = env["tmp"] / "proj"
     _write_mise(root, with_test=False)
     _register_project(root)
     with pytest.raises(spawn.SpawnError, match="no \\[tasks.test\\]"):
-        spawn.spawn("proj", "api")
+        spawn.spawn("proj", "api", task="do it")
-    assert fake_tmux["calls"]["new_session"] == []
+    assert fake_launch == []
 
 
-def test_spawn_refuses_without_mise_file(env, fake_tmux):
+def test_spawn_refuses_without_mise_file(env, fake_launch):
     root = env["tmp"] / "proj"
     root.mkdir(parents=True, exist_ok=True)
     _register_project(root)
     with pytest.raises(spawn.SpawnError, match="no mise config"):
+        spawn.spawn("proj", "api", task="do it")
+
+
+def test_spawn_refuses_without_task(env, fake_launch):
+    root = env["tmp"] / "proj"
+    _write_mise(root)
+    _register_project(root)
+    with pytest.raises(spawn.SpawnError, match="requires a task"):
         spawn.spawn("proj", "api")
+    # Fail-fast: no orphaned agent row behind the refused spawn.
+    with get_engine().begin() as conn:
+        assert repo.get_agent_by_name(conn, "proj", "api") is None
+    assert fake_launch == []
 
 
-def test_spawn_accepts_dotless_mise_toml(env, fake_tmux):
+def test_spawn_accepts_dotless_mise_toml(env, fake_launch):
     # mise also reads `mise.toml` (no leading dot); the gate must honor it too.
     root = env["tmp"] / "proj"
     root.mkdir(parents=True, exist_ok=True)
     (root / "mise.toml").write_text("[tasks.test]\nrun = 'pytest'\n")
     _register_project(root)
 
-    agent = spawn.spawn("proj", "api")
+    agent = spawn.spawn("proj", "api", task="do it")
     with get_engine().begin() as conn:
         assert repo.get_agent_by_name(conn, "proj", "api")["id"] == agent["id"]
 
 
-def test_spawn_creates_agent_settings_and_session(env, fake_tmux):
+def test_spawn_creates_agent_settings_and_run(env, fake_launch):
     root = env["tmp"] / "proj"
     _write_mise(root, with_test=True)
     _register_project(root)
@@ -64,66 +79,99 @@ def test_spawn_creates_agent_settings_and_session(env, fake_tmux):
     with get_engine().begin() as conn:
         assert repo.get_agent_by_name(conn, "proj", "api")["id"] == agent["id"]
 
-    # settings.json wires all four hook events.
+    # settings.json wires all four hook events AND the headless permission allowlist
+    # (claude -p auto-denies anything that would prompt; the allowlist is what lets
+    # normal work proceed — the hooks stay the hard gate).
     settings = json.loads((root / ".claude" / "settings.json").read_text())
     assert set(settings["hooks"]) == {"Stop", "SessionEnd", "PreToolUse", "Notification"}
     pre = settings["hooks"]["PreToolUse"][0]
     assert pre["matcher"] == "AskUserQuestion|Bash"
     assert "handler.hooks pre_tool_use" in pre["hooks"][0]["command"]
+    assert settings["permissions"]["defaultMode"] == "acceptEdits"
+    assert "Bash(git *)" in settings["permissions"]["allow"]
 
-    # tmux session named project__agent, with identity + DATABASE_URL in env.
+    # A headless run launched with identity + DATABASE_URL in env and the task as prompt.
-    call = fake_tmux["calls"]["new_session"][0]
+    call = fake_launch[0]
-    assert call["name"] == "proj__api"
+    assert call["kind"] == "spawn"
+    assert call["prompt"] == "build the thing"
     assert call["env"]["HANDLER_PROJECT_ID"] == "proj"
     assert call["env"]["HANDLER_AGENT_NAME"] == "api"
     assert call["env"]["HANDLER_AGENT_ID"] == str(agent["id"])
     assert call["env"]["DATABASE_URL"] == env["url"]
+    # The run row + session id landed on the agent.
+    with get_engine().begin() as conn:
+        row = repo.get_agent_by_name(conn, "proj", "api")
+        assert row["session_id"] == call["run"]["session_id"]
+        assert repo.get_latest_run(conn, row["id"])["kind"] == "spawn"
 
 
-def test_spawn_mise_init_skips_test_gate_and_marks_env(env, fake_tmux):
+def test_spawn_mise_init_skips_test_gate_and_marks_env(env, fake_launch):
     # A repo with no .mise.toml at all: the normal gate would refuse, but the mise-init
     # bootstrap agent must launch anyway (creating that file is its whole job).
     root = env["tmp"] / "proj"
     root.mkdir(parents=True, exist_ok=True)
     _register_project(root)
 
-    agent = spawn.spawn("proj", "mise-init", require_tests=False, mise_init=True)
+    agent = spawn.spawn(
+        "proj", "mise-init", task="write the mise config", require_tests=False, mise_init=True
+    )
 
     with get_engine().begin() as conn:
         assert repo.get_agent_by_name(conn, "proj", "mise-init")["id"] == agent["id"]
-    # The launched session carries HANDLER_MISE_INIT so its hooks enforce commit + push.
+    # The launched run carries HANDLER_MISE_INIT so its hooks enforce commit + push.
-    call = fake_tmux["calls"]["new_session"][0]
+    assert fake_launch[0]["env"]["HANDLER_MISE_INIT"] == "1"
-    assert call["env"]["HANDLER_MISE_INIT"] == "1"
 
 
-def test_spawn_still_gates_without_mise_init_flag(env, fake_tmux):
+def test_spawn_still_gates_without_mise_init_flag(env, fake_launch):
     root = env["tmp"] / "proj"
     root.mkdir(parents=True, exist_ok=True)
     _register_project(root)
     # require_tests defaults on, so a normal spawn against a mise-less repo still refuses.
     with pytest.raises(spawn.SpawnError, match="no mise config"):
-        spawn.spawn("proj", "api")
+        spawn.spawn("proj", "api", task="do it")
-    assert fake_tmux["calls"]["new_session"] == []
+    assert fake_launch == []
 
 
-def test_kill_sets_done_and_kills_session(env, fake_tmux):
+def test_kill_cancels_run_and_sets_done(env, fake_launch):
     root = env["tmp"] / "proj"
     _write_mise(root, with_test=True)
     _register_project(root)
-    spawn.spawn("proj", "api")
+    spawn.spawn("proj", "api", task="do it")
 
     spawn.kill("proj", "api")
-    assert "proj__api" in fake_tmux["calls"]["kill_session"]
     with get_engine().begin() as conn:
-        assert repo.get_agent_by_name(conn, "proj", "api")["status"] == "done"
+        agent = repo.get_agent_by_name(conn, "proj", "api")
+        assert agent["status"] == "done"
+        # The running run was flagged; the owning supervisor terminates its own child.
+        assert repo.get_latest_run(conn, agent["id"])["cancel_requested"] is True
 
 
-def test_resume_sends_answer_to_live_session(env, fake_tmux):
+def test_resume_reinjects_when_no_transcript(env, fake_launch):
+    """A resume with no archive and no local transcript degrades to a fresh run whose
+    prompt carries the operator's answer (context re-injection)."""
     root = env["tmp"] / "proj"
     _write_mise(root, with_test=True)
     _register_project(root)
-    agent = spawn.spawn("proj", "api")
+    spawn.spawn("proj", "api", task="do it")
+    with get_engine().begin() as conn:
+        agent = repo.get_agent_by_name(conn, "proj", "api")
+        repo.finish_run(conn, repo.get_latest_run(conn, agent["id"])["id"], "completed")
 
     ok, detail = spawn.resume(agent, "use Postgres")
     assert ok is True
-    assert fake_tmux["calls"]["send_keys"][0] == {"name": "proj__api", "keys": "use Postgres"}
+    assert "re-injected" in detail
+    assert fake_launch[-1]["kind"] == "spawn"
+    assert "use Postgres" in fake_launch[-1]["prompt"]
+
+
+def test_resume_refused_while_run_live(env, fake_launch):
+    root = env["tmp"] / "proj"
+    _write_mise(root, with_test=True)
+    _register_project(root)
+    spawn.spawn("proj", "api", task="do it")  # fake run stays 'running'
+    with get_engine().begin() as conn:
+        agent = repo.get_agent_by_name(conn, "proj", "api")
+
+    ok, detail = spawn.resume(agent, "answer")
+    assert ok is False
+    assert "live run" in detail

tests/test_control_spawn_forge.py

@@ -19,15 +19,15 @@ def _register(root, **kw):
         repo.create_project(conn, "proj", str(root), **kw)
 
 
-def test_spawn_injects_credentials_and_installs_helper(env, fake_tmux, fake_gitops, monkeypatch):
+def test_spawn_injects_credentials_and_installs_helper(env, fake_launch, fake_gitops, monkeypatch):
     monkeypatch.setenv("PROJ_TOKEN", "s3cret")
     root = env["tmp"] / "proj"
     _write_mise(root)
     _register(root, git_remote="https://github.com/me/proj.git", credential_ref="env:PROJ_TOKEN")
 
-    spawn.spawn("proj", "junior", role="junior")
+    spawn.spawn("proj", "junior", role="junior", task="do it")
 
-    call = fake_tmux["calls"]["new_session"][0]
+    call = fake_launch[0]
     # Token injected under the generic + host-specific names, never the raw ref stored.
     assert call["env"]["FORGE_TOKEN"] == "s3cret"
     assert call["env"]["GITHUB_TOKEN"] == "s3cret"
@@ -38,43 +38,43 @@ def test_spawn_injects_credentials_and_installs_helper(env, fake_tmux, fake_gito
     assert "$FORGE_TOKEN" in helper[0]["value"]
 
 
-def test_spawn_ssh_remote_installs_no_https_helper(env, fake_tmux, fake_gitops, monkeypatch):
+def test_spawn_ssh_remote_installs_no_https_helper(env, fake_launch, fake_gitops, monkeypatch):
     monkeypatch.setenv("PROJ_TOKEN", "s3cret")
     root = env["tmp"] / "proj"
     _write_mise(root)
     _register(root, git_remote="git@github.com:me/proj.git", credential_ref="env:PROJ_TOKEN")
-    spawn.spawn("proj", "junior", role="junior")
+    spawn.spawn("proj", "junior", role="junior", task="do it")
     # ssh remote -> token still injected, but no HTTPS credential helper installed.
-    assert fake_tmux["calls"]["new_session"][0]["env"]["GITHUB_TOKEN"] == "s3cret"
+    assert fake_launch[0]["env"]["GITHUB_TOKEN"] == "s3cret"
     assert fake_gitops["config"] == []
 
 
-def test_spawn_fails_fast_on_broken_credential_ref(env, fake_tmux, fake_gitops, monkeypatch):
+def test_spawn_fails_fast_on_broken_credential_ref(env, fake_launch, fake_gitops, monkeypatch):
     monkeypatch.delenv("ABSENT_TOKEN", raising=False)
     root = env["tmp"] / "proj"
     _write_mise(root)
     _register(root, credential_ref="env:ABSENT_TOKEN")
 
     with pytest.raises(spawn.SpawnError, match="not set"):
-        spawn.spawn("proj", "junior", role="junior")
+        spawn.spawn("proj", "junior", role="junior", task="do it")
     # No agent row and no session left behind by the failed spawn.
     with get_engine().begin() as conn:
         assert repo.get_agent_by_name(conn, "proj", "junior") is None
-    assert fake_tmux["calls"]["new_session"] == []
+    assert fake_launch == []
 
 
-def test_spawn_without_credential_ref_injects_no_token(env, fake_tmux, fake_gitops):
+def test_spawn_without_credential_ref_injects_no_token(env, fake_launch, fake_gitops):
     root = env["tmp"] / "proj"
     _write_mise(root)
     _register(root)
-    spawn.spawn("proj", "api")
+    spawn.spawn("proj", "api", task="do it")
-    call = fake_tmux["calls"]["new_session"][0]
+    call = fake_launch[0]
     assert "FORGE_TOKEN" not in call["env"]
     # No token -> no credential helper installed.
     assert fake_gitops["config"] == []
 
 
-def test_spawn_reports_forge_version_mismatch(env, fake_tmux, fake_gitops, fake_forge, monkeypatch):
+def test_spawn_reports_forge_version_mismatch(env, fake_launch, fake_gitops, fake_forge, monkeypatch):
     monkeypatch.setenv("FORGE_VERSION", "9.9.9")
     from handler import config
     from handler.db import engine
@@ -88,5 +88,5 @@ def test_spawn_reports_forge_version_mismatch(env, fake_tmux, fake_gitops, fake_
     fake_forge["version_ok"] = False
     fake_forge["version_out"] = "forge 1.2.3"
 
-    agent = spawn.spawn("proj", "api")
+    agent = spawn.spawn("proj", "api", task="do it")
     assert "9.9.9" in agent["forge_note"]

tests/test_credsync.py

@@ -0,0 +1,103 @@
+"""Credential distribution through runtime_secrets: the login completes on one worker,
+every other worker materializes the encrypted bundle from the DB — no shared files."""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+
+import pytest
+from cryptography.fernet import Fernet
+
+from handler.control import credsync
+from handler.db import repository as repo
+from handler.db.engine import get_engine
+
+
+@pytest.fixture
+def secret_env(env, monkeypatch):
+    from handler import config
+
+    monkeypatch.setenv("HANDLER_SECRET_KEY", Fernet.generate_key().decode())
+    config.get_settings.cache_clear()
+    credsync._state.__init__()  # fresh sync cursor per test
+    yield env
+    config.get_settings.cache_clear()
+
+
+def _write_local_credentials(home: Path) -> None:
+    (home / ".claude").mkdir(parents=True, exist_ok=True)
+    (home / ".claude.json").write_text(
+        json.dumps({"oauthAccount": {"email": "op@example.com"}, "theme": "light"})
+    )
+    (home / ".claude" / ".credentials.json").write_text('{"token": "secret-oauth-token"}')
+
+
+def test_upload_stores_encrypted_bundle(secret_env, tmp_path):
+    _write_local_credentials(tmp_path)
+    assert credsync.upload() is True
+    with get_engine().begin() as conn:
+        row = repo.get_runtime_secret(conn, credsync.SECRET_KEY)
+    assert row is not None
+    # Ciphertext at rest — the raw token must not appear in the DB value.
+    assert "secret-oauth-token" not in row["value_enc"]
+
+
+def test_refresh_materializes_on_fresh_worker(secret_env, tmp_path, monkeypatch):
+    _write_local_credentials(tmp_path)
+    assert credsync.upload() is True
+
+    other_home = tmp_path / "worker-b"
+    other_home.mkdir()
+    monkeypatch.setenv("HOME", str(other_home))
+    credsync._state.__init__()  # worker B's process state
+
+    assert credsync.refresh() == "materialized"
+    creds = json.loads((other_home / ".claude" / ".credentials.json").read_text())
+    assert creds["token"] == "secret-oauth-token"
+    data = json.loads((other_home / ".claude.json").read_text())
+    assert data["oauthAccount"]["email"] == "op@example.com"
+    # A second pass is a no-op — nothing changed anywhere.
+    assert credsync.refresh() is None
+
+
+def test_materialize_merges_claude_json_preserving_local_state(secret_env, tmp_path, monkeypatch):
+    _write_local_credentials(tmp_path)
+    credsync.upload()
+
+    other_home = tmp_path / "worker-c"
+    (other_home / ".claude").mkdir(parents=True)
+    (other_home / ".claude.json").write_text(
+        json.dumps(
+            {
+                "hasCompletedOnboarding": True,
+                "theme": "dark",
+                "projects": {"/projects/p/a": {"hasTrustDialogAccepted": True}},
+            }
+        )
+    )
+    monkeypatch.setenv("HOME", str(other_home))
+    credsync._state.__init__()
+
+    assert credsync.refresh() == "materialized"
+    data = json.loads((other_home / ".claude.json").read_text())
+    # Account arrived...
+    assert data["oauthAccount"]["email"] == "op@example.com"
+    # ...but this worker's own onboarding/trust state (claude_config's writes) survived.
+    assert data["theme"] == "dark"
+    assert data["projects"]["/projects/p/a"]["hasTrustDialogAccepted"] is True
+
+
+def test_refresh_uploads_local_change(secret_env, tmp_path):
+    _write_local_credentials(tmp_path)
+    assert credsync.refresh() == "uploaded"  # bootstrap: local creds, empty DB
+    # A token refresh on disk (mtime/size change) re-publishes.
+    os.utime(tmp_path / ".claude" / ".credentials.json", ns=(1, 1))
+    assert credsync.refresh() == "uploaded"
+
+
+def test_disabled_without_secret_key(env, tmp_path):
+    _write_local_credentials(tmp_path)
+    assert credsync.upload() is False
+    assert credsync.refresh() is None

tests/test_headless_parser.py

@@ -0,0 +1,178 @@
+"""Pure-helper tests for the headless runner: stream parsing, path munging, argv
+construction, and the session archive round-trip. No subprocess is launched here —
+the process-level tests live in test_headless_run.py (phase 2)."""
+
+from __future__ import annotations
+
+import io
+import tarfile
+
+from handler.control import headless
+
+
+# ------------------------------------------------------------------ parse_stream_line
+
+
+def test_parse_valid_event_types():
+    for etype in ("system", "assistant", "user", "result", "hook"):
+        parsed_type, payload = headless.parse_stream_line(f'{{"type": "{etype}", "x": 1}}\n')
+        assert parsed_type == etype
+        assert payload == {"type": etype, "x": 1}
+
+
+def test_parse_unknown_type_keeps_its_name():
+    parsed_type, payload = headless.parse_stream_line('{"type": "telemetry", "n": 2}')
+    assert parsed_type == "telemetry"
+    assert payload["n"] == 2
+
+
+def test_parse_garbage_becomes_raw():
+    parsed_type, payload = headless.parse_stream_line("this is not json\n")
+    assert parsed_type == "raw"
+    assert payload == {"line": "this is not json\n"}
+
+
+def test_parse_non_dict_json_becomes_raw():
+    parsed_type, payload = headless.parse_stream_line('["a", "b"]')
+    assert parsed_type == "raw"
+
+
+def test_parse_missing_type_becomes_raw():
+    parsed_type, payload = headless.parse_stream_line('{"message": "no type field"}')
+    assert parsed_type == "raw"
+    assert payload == {"message": "no type field"}
+
+
+def test_parse_blank_line_becomes_raw():
+    parsed_type, _ = headless.parse_stream_line("   \n")
+    assert parsed_type == "raw"
+
+
+# -------------------------------------------------------------------- assistant_text
+
+
+def test_assistant_text_joins_text_blocks():
+    payload = {
+        "type": "assistant",
+        "message": {
+            "content": [
+                {"type": "text", "text": "first"},
+                {"type": "tool_use", "name": "Bash", "input": {}},
+                {"type": "text", "text": "second"},
+            ]
+        },
+    }
+    assert headless.assistant_text(payload) == "first\nsecond"
+
+
+def test_assistant_text_none_for_pure_tool_use():
+    payload = {
+        "type": "assistant",
+        "message": {"content": [{"type": "tool_use", "name": "Bash", "input": {}}]},
+    }
+    assert headless.assistant_text(payload) is None
+
+
+def test_assistant_text_string_content():
+    assert headless.assistant_text({"message": {"content": "plain"}}) == "plain"
+
+
+def test_assistant_text_missing_message():
+    assert headless.assistant_text({"type": "assistant"}) is None
+
+
+# ---------------------------------------------------------------- munged_project_dir
+
+
+def test_munge_matches_recorded_real_examples():
+    # Recorded from a real ~/.claude/projects/ (see plan): '/' and '.' both map to '-'.
+    assert headless.munged_project_dir("/root/handler") == "-root-handler"
+    assert (
+        headless.munged_project_dir("/root/Talos/.claude/worktrees/mise-tooling")
+        == "-root-Talos--claude-worktrees-mise-tooling"
+    )
+
+
+# ------------------------------------------------------------------- argv builders
+
+
+def test_spawn_argv_shape(env):
+    argv = headless.build_spawn_argv("do the task", "/wd/.claude/settings.json", "sid-1")
+    assert argv[0] == "claude"
+    assert "-p" in argv and "--verbose" in argv
+    assert argv[argv.index("--output-format") + 1] == "stream-json"
+    assert argv[argv.index("--session-id") + 1] == "sid-1"
+    assert argv[argv.index("--settings") + 1] == "/wd/.claude/settings.json"
+    assert "--max-budget-usd" not in argv  # default budget is 0 = off
+    assert argv[-2:] == ["--", "do the task"]
+    assert "--resume" not in argv
+
+
+def test_resume_argv_shape(env):
+    argv = headless.build_resume_argv("sid-2", "the answer", "/wd/.claude/settings.json")
+    assert argv[argv.index("--resume") + 1] == "sid-2"
+    assert argv[-2:] == ["--", "the answer"]
+    assert "--session-id" not in argv
+
+
+def test_spawn_argv_includes_budget_when_set(env, monkeypatch):
+    monkeypatch.setenv("RUN_BUDGET_USD", "2.5")
+    from handler import config
+
+    config.get_settings.cache_clear()
+    argv = headless.build_spawn_argv("t", "/s.json", "sid")
+    assert argv[argv.index("--max-budget-usd") + 1] == "2.5"
+    config.get_settings.cache_clear()
+
+
+# ------------------------------------------------------------- archive round-trip
+
+
+def _write_fake_session(home, working_dir: str, session_id: str) -> None:
+    base = home / ".claude" / "projects" / headless.munged_project_dir(working_dir)
+    base.mkdir(parents=True)
+    (base / f"{session_id}.jsonl").write_text('{"type": "user", "prompt": "hi"}\n')
+    sidecar = base / session_id / "tool-results"
+    sidecar.mkdir(parents=True)
+    (sidecar / "r1.txt").write_text("tool output")
+
+
+def test_archive_and_materialize_round_trip(env, tmp_path, monkeypatch):
+    working_dir = "/projects/demo"
+    _write_fake_session(tmp_path, working_dir, "sid-rt")
+
+    data = headless.archive_session(working_dir, "sid-rt")
+    assert data is not None
+
+    # Materialize onto a *different* worker: a fresh HOME with no session state.
+    other_home = tmp_path / "other-worker"
+    other_home.mkdir()
+    monkeypatch.setenv("HOME", str(other_home))
+    headless.materialize_session(working_dir, data)
+
+    base = other_home / ".claude" / "projects" / headless.munged_project_dir(working_dir)
+    assert (base / "sid-rt.jsonl").read_text() == '{"type": "user", "prompt": "hi"}\n'
+    assert (base / "sid-rt" / "tool-results" / "r1.txt").read_text() == "tool output"
+
+
+def test_archive_none_when_no_session(env):
+    assert headless.archive_session("/projects/none", "missing-sid") is None
+
+
+def test_archive_refuses_oversize(env, tmp_path):
+    working_dir = "/projects/big"
+    _write_fake_session(tmp_path, working_dir, "sid-big")
+    assert headless.archive_session(working_dir, "sid-big", max_bytes=10) is None
+
+
+def test_archive_contains_only_session_members(env, tmp_path):
+    working_dir = "/projects/demo2"
+    _write_fake_session(tmp_path, working_dir, "sid-a")
+    # A sibling session must not leak into sid-a's archive.
+    base = tmp_path / ".claude" / "projects" / headless.munged_project_dir(working_dir)
+    (base / "sid-other.jsonl").write_text("{}\n")
+
+    data = headless.archive_session(working_dir, "sid-a")
+    with tarfile.open(fileobj=io.BytesIO(data), mode="r:gz") as tar:
+        names = tar.getnames()
+    assert all(n == "sid-a.jsonl" or n.startswith("sid-a") for n in names)

tests/test_headless_run.py

@@ -0,0 +1,268 @@
+"""End-to-end headless runner tests against the fake ``claude`` binary.
+
+Real subprocesses, real threads, real SQLite: ``headless.launch`` starts
+``tests/fixtures/fake_claude.py`` (selected via the ``claude_bin`` setting, the same
+seam the tmux fakes used), the supervisor streams its stdout into ``agent_events``, and
+the tests assert on what landed in the DB — exactly what the API/UI will read."""
+
+from __future__ import annotations
+
+import json
+import time
+from pathlib import Path
+
+import pytest
+
+from handler.control import headless, settings_gen, spawn
+from handler.db import repository as repo
+from handler.db.engine import get_engine
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+FAKE_CLAUDE = str(REPO_ROOT / "tests" / "fixtures" / "fake_claude.py")
+
+
+@pytest.fixture
+def headless_env(env, monkeypatch):
+    from handler import config
+
+    monkeypatch.setenv("CLAUDE_BIN", FAKE_CLAUDE)
+    config.get_settings.cache_clear()
+    yield env
+    config.get_settings.cache_clear()
+
+
+def _make_agent(tmp_path, name="h1"):
+    working_dir = tmp_path / "projects" / "p" / name
+    working_dir.mkdir(parents=True)
+    with get_engine().begin() as conn:
+        repo.create_project(conn, "p", str(tmp_path / "projects" / "p"))
+        agent = repo.create_agent(conn, "p", name, str(working_dir))
+    return agent
+
+
+def _wait_for(predicate, timeout=20.0):
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        result = predicate()
+        if result:
+            return result
+        time.sleep(0.1)
+    return None
+
+
+def _finished_run(run_id):
+    def check():
+        with get_engine().begin() as conn:
+            run = repo.get_run(conn, run_id)
+        return run if run["status"] != "running" else None
+
+    return check
+
+
+def test_spawn_run_streams_events_and_completes(headless_env, tmp_path):
+    agent = _make_agent(tmp_path)
+    run = headless.launch(
+        agent, kind="spawn", prompt="build the thing",
+        settings_path=str(tmp_path / "settings.json"), env={}, worker_id="w1",
+    )
+    finished = _wait_for(_finished_run(run["id"]))
+    assert finished is not None, "run never finished"
+    assert finished["status"] == "completed"
+    assert finished["exit_code"] == 0
+    assert finished["result"]["is_error"] is False
+
+    with get_engine().begin() as conn:
+        events = repo.list_agent_events(conn, agent["id"])
+        updated = repo.get_agent_by_id(conn, agent["id"])
+        archive = repo.get_session_archive(conn, agent["id"])
+
+    types = [e["type"] for e in events]
+    assert types == ["system", "assistant", "result"]
+    assert [e["seq"] for e in events] == [1, 2, 3]
+    # last_output is now derived from assistant text — the log the UI shows is real.
+    assert updated["last_output"] == "working on: build the thing"
+    assert updated["status"] == "done"
+    assert updated["session_id"] == run["session_id"]
+    assert updated["worker_id"] == "w1"
+    # The session archive was uploaded at exit for cross-worker resume.
+    assert archive is not None
+    assert archive["session_id"] == run["session_id"]
+
+
+def test_failed_run_marks_blocked_with_worker_event(headless_env, tmp_path, monkeypatch):
+    monkeypatch.setenv("FAKE_CLAUDE_MODE", "error")
+    agent = _make_agent(tmp_path, "h-err")
+    run = headless.launch(
+        agent, kind="spawn", prompt="boom",
+        settings_path=str(tmp_path / "s.json"), env={}, worker_id="w1",
+    )
+    finished = _wait_for(_finished_run(run["id"]))
+    assert finished["status"] == "failed"
+    assert finished["exit_code"] == 2
+
+    with get_engine().begin() as conn:
+        events = repo.list_agent_events(conn, agent["id"])
+        updated = repo.get_agent_by_id(conn, agent["id"])
+    types = [e["type"] for e in events]
+    # The unparseable stdout line is preserved verbatim as a raw event, and the runner
+    # records why the run failed as a worker event.
+    assert "raw" in types
+    raw = next(e for e in events if e["type"] == "raw")
+    assert "this is not json" in raw["payload"]["line"]
+    worker_ev = next(e for e in events if e["type"] == "worker")
+    assert worker_ev["payload"]["exit_code"] == 2
+    assert updated["status"] == "blocked"
+
+
+def test_hook_written_status_survives_reconciliation(headless_env, tmp_path, monkeypatch):
+    """Hooks are the status authority: if one set paused_for_input during the run, the
+    supervisor's exit pass must not overwrite it with done."""
+    monkeypatch.setenv("FAKE_CLAUDE_MODE", "slow")
+    monkeypatch.setenv("FAKE_CLAUDE_SLOW_SECONDS", "1.5")
+    agent = _make_agent(tmp_path, "h-hook")
+    run = headless.launch(
+        agent, kind="spawn", prompt="ask me something",
+        settings_path=str(tmp_path / "s.json"), env={}, worker_id="w1",
+    )
+    # Simulate a hook (inside the run) recording an open question.
+    with get_engine().begin() as conn:
+        repo.set_agent_status(conn, agent["id"], "paused_for_input")
+    finished = _wait_for(_finished_run(run["id"]))
+    assert finished["status"] == "completed"
+    with get_engine().begin() as conn:
+        assert repo.get_agent_by_id(conn, agent["id"])["status"] == "paused_for_input"
+
+
+def test_cancel_terminates_hanging_run(headless_env, tmp_path, monkeypatch):
+    monkeypatch.setenv("FAKE_CLAUDE_MODE", "hang")
+    agent = _make_agent(tmp_path, "h-hang")
+    run = headless.launch(
+        agent, kind="spawn", prompt="hang forever",
+        settings_path=str(tmp_path / "s.json"), env={}, worker_id="w1",
+    )
+    # Give the supervisor a moment to start the process, then flag the cancel the same
+    # way a cross-worker kill would.
+    _wait_for(lambda: _events_count(agent["id"]) >= 1)
+    with get_engine().begin() as conn:
+        assert repo.request_run_cancel(conn, run["id"]) is True
+    finished = _wait_for(_finished_run(run["id"]), timeout=30.0)
+    assert finished is not None, "cancel never terminated the run"
+    assert finished["status"] == "canceled"
+
+
+def _events_count(agent_id):
+    with get_engine().begin() as conn:
+        return len(repo.list_agent_events(conn, agent_id))
+
+
+def test_kill_headless_agent_requests_cancel(headless_env, tmp_path, monkeypatch):
+    monkeypatch.setenv("FAKE_CLAUDE_MODE", "hang")
+    agent = _make_agent(tmp_path, "h-kill")
+    run = headless.launch(
+        agent, kind="spawn", prompt="hang",
+        settings_path=str(tmp_path / "s.json"), env={}, worker_id="w1",
+    )
+    _wait_for(lambda: _events_count(agent["id"]) >= 1)
+    spawn.kill("p", "h-kill")
+    finished = _wait_for(_finished_run(run["id"]), timeout=30.0)
+    assert finished["status"] == "canceled"
+    with get_engine().begin() as conn:
+        assert repo.get_agent_by_id(conn, agent["id"])["status"] == "done"
+
+
+def test_cross_worker_resume_materializes_archive(headless_env, tmp_path, monkeypatch):
+    """The linchpin: worker B resumes a session it never ran, from the DB archive alone."""
+    agent = _make_agent(tmp_path, "h-resume")
+    run = headless.launch(
+        agent, kind="spawn", prompt="first pass",
+        settings_path=str(tmp_path / "s.json"), env={}, worker_id="worker-a",
+    )
+    assert _wait_for(_finished_run(run["id"]))["status"] == "completed"
+
+    # "Worker B": a clean HOME with no local claude state at all.
+    other_home = tmp_path / "worker-b-home"
+    other_home.mkdir()
+    monkeypatch.setenv("HOME", str(other_home))
+
+    with get_engine().begin() as conn:
+        agent = repo.get_agent_by_id(conn, agent["id"])  # refetch: has session_id now
+    ok, detail = spawn.resume(agent, "the operator's answer", worker_id="worker-b")
+    assert ok, detail
+
+    with get_engine().begin() as conn:
+        resumed = repo.get_latest_run(conn, agent["id"])
+    assert resumed["kind"] == "resume"
+    assert resumed["worker_id"] == "worker-b"
+    finished = _wait_for(_finished_run(resumed["id"]))
+    # fake_claude exits 3 if the transcript was NOT materialized where claude looks.
+    assert finished["status"] == "completed", f"exit={finished['exit_code']}"
+    assert finished["session_id"] == run["session_id"]  # same session, continued
+
+
+def test_resume_without_transcript_reinjects_context(headless_env, tmp_path, monkeypatch):
+    """Owning worker died before its first archive: resume degrades to a fresh session
+    with DB-rebuilt context, visibly marked as such."""
+    agent = _make_agent(tmp_path, "h-fallback")
+    with get_engine().begin() as conn:
+        repo.set_agent_session(conn, agent["id"], "lost-session-uuid", "worker-dead")
+        repo.upsert_checkmark_row(
+            conn, agent["id"], status="paused_for_input",
+            where_it_stopped="mid-refactor", open_question="which db?",
+        )
+        agent = repo.get_agent_by_id(conn, agent["id"])
+
+    ok, detail = spawn.resume(agent, "use postgres", worker_id="worker-b")
+    assert ok
+    assert "re-injected" in detail
+
+    with get_engine().begin() as conn:
+        new_run = repo.get_latest_run(conn, agent["id"])
+        events = repo.list_agent_events(conn, agent["id"])
+    assert new_run["kind"] == "spawn"  # genuinely new session
+    assert new_run["session_id"] != "lost-session-uuid"
+    notice = next(e for e in events if e["type"] == "worker")
+    assert notice["payload"]["previous_session_id"] == "lost-session-uuid"
+    finished = _wait_for(_finished_run(new_run["id"]))
+    assert finished["status"] == "completed"
+    with get_engine().begin() as conn:
+        assistant = [
+            e for e in repo.list_agent_events(conn, agent["id"]) if e["type"] == "assistant"
+        ]
+    # The re-injected prompt (with the operator's answer) reached the fresh claude.
+    assert any("use postgres" in json.dumps(e["payload"]) for e in assistant)
+
+
+def test_resume_refused_while_run_live(headless_env, tmp_path, monkeypatch):
+    monkeypatch.setenv("FAKE_CLAUDE_MODE", "hang")
+    agent = _make_agent(tmp_path, "h-busy")
+    run = headless.launch(
+        agent, kind="spawn", prompt="hang",
+        settings_path=str(tmp_path / "s.json"), env={}, worker_id="w1",
+    )
+    with get_engine().begin() as conn:
+        agent = repo.get_agent_by_id(conn, agent["id"])
+    ok, detail = spawn.resume(agent, "answer", worker_id="w1")
+    assert not ok
+    assert "live run" in detail
+    with get_engine().begin() as conn:
+        repo.request_run_cancel(conn, run["id"])
+    _wait_for(_finished_run(run["id"]), timeout=30.0)
+
+
+def test_settings_include_permissions_and_hooks(headless_env, tmp_path):
+    path = settings_gen.write_settings(str(tmp_path / "wd"))
+    data = json.loads(Path(path).read_text())
+    assert data["permissions"]["defaultMode"] == "acceptEdits"
+    assert "Bash(git *)" in data["permissions"]["allow"]
+    assert "hooks" in data  # the hard gate is untouched
+
+
+def test_api_rejects_empty_task_headless_spawn(headless_env, client, auth):
+    client.post(
+        "/projects", json={"id": "p2", "root_dir": "/tmp/p2"}, headers=auth
+    )
+    resp = client.post(
+        "/projects/p2/agents/spawn", json={"name": "idle"}, headers=auth
+    )
+    assert resp.status_code == 400
+    assert "task is required" in resp.json()["detail"]

tests/test_integration_web_spawn.py

@@ -1,13 +1,32 @@
 """End-to-end web management: the dashboard's HTTP calls -> command queue -> worker ->
-real ``spawn.spawn`` -> tmux seam. Proves the full container-split flow works with only the
+real ``spawn.spawn`` -> a real headless subprocess (the fake claude binary). Proves the
-tmux/claude boundary faked, not the control layer itself."""
+full container-split flow works with only the claude binary faked, not the control
+layer: events stream into the DB, the run reconciles, kill cancels."""
 
 from __future__ import annotations
 
+import time
+from pathlib import Path
+
+import pytest
+
 from handler.control import worker
 from handler.db import repository as repo
 from handler.db.engine import get_engine
 
+REPO_ROOT = Path(__file__).resolve().parents[1]
+FAKE_CLAUDE = str(REPO_ROOT / "tests" / "fixtures" / "fake_claude.py")
+
+
+@pytest.fixture
+def headless_env(env, monkeypatch):
+    from handler import config
+
+    monkeypatch.setenv("CLAUDE_BIN", FAKE_CLAUDE)
+    config.get_settings.cache_clear()
+    yield env
+    config.get_settings.cache_clear()
+
 
 def _spawnable_project(root):
     root.mkdir(parents=True, exist_ok=True)
@@ -16,10 +35,21 @@ def _spawnable_project(root):
         repo.create_project(conn, "proj", str(root))
 
 
-def test_spawn_via_api_then_worker_creates_agent_and_session(client, auth, env, fake_tmux):
+def _wait(predicate, timeout=20.0):
-    _spawnable_project(env["tmp"] / "proj")
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        result = predicate()
+        if result:
+            return result
+        time.sleep(0.1)
+    return None
 
-    # 1. The dashboard enqueues a spawn (202 + a queued command).
+
+def test_spawn_via_api_then_worker_runs_headless_claude(client, auth, headless_env):
+    _spawnable_project(headless_env["tmp"] / "proj")
+
+    # 1. The dashboard enqueues a spawn (202 + a queued command). A task is mandatory —
+    #    headless claude has no idle-REPL mode.
     r = client.post(
         "/projects/proj/agents/spawn",
         json={"name": "api", "task": "build the thing"},
@@ -32,29 +62,62 @@ def test_spawn_via_api_then_worker_creates_agent_and_session(client, auth, env,
     # No agent yet — the worker hasn't run.
     assert client.get("/projects/proj/agents", headers=auth).json() == []
 
-    # 2. The control worker drains the queue (runs the real spawn.spawn).
+    # 2. The control worker drains the queue (real spawn.spawn -> real subprocess).
     assert worker.drain("test-worker") == 1
 
-    # 3. The command is done and the agent + tmux session now exist.
+    # 3. The command finished at launch (fire-and-forget)...
     got = client.get(f"/commands/{command_id}", headers=auth).json()
     assert got["status"] == "done"
     assert got["result"]["name"] == "api"
 
-    agents = client.get("/projects/proj/agents", headers=auth).json()
+    # ...and the run's whole life shows up via the API: events stream in, the agent
-    assert [a["name"] for a in agents] == ["api"]
+    # reconciles to done, last_output is the assistant's text.
-    assert fake_tmux["calls"]["new_session"][0]["name"] == "proj__api"
+    def finished():
+        agents = client.get("/projects/proj/agents", headers=auth).json()
+        return agents if agents and agents[0]["status"] == "done" else None
+
+    agents = _wait(finished)
+    assert agents is not None, "run never reconciled to done"
+    agent = agents[0]
+    assert agent["name"] == "api"
+    assert agent["session_id"]
+    assert agent["worker_id"] == "test-worker"
+    assert agent["last_output"] == "working on: build the thing"
+
+    events = client.get("/projects/proj/agents/api/events", headers=auth).json()
+    assert [e["type"] for e in events] == ["system", "assistant", "result"]
 
 
-def test_kill_via_api_then_worker(client, auth, env, fake_tmux):
+def test_spawn_without_task_is_rejected(client, auth, headless_env):
-    _spawnable_project(env["tmp"] / "proj")
+    _spawnable_project(headless_env["tmp"] / "proj")
-    client.post("/projects/proj/agents/spawn", json={"name": "api"}, headers=auth)
+    r = client.post("/projects/proj/agents/spawn", json={"name": "api"}, headers=auth)
+    assert r.status_code == 400
+    assert "task is required" in r.json()["detail"]
+
+
+def test_kill_via_api_then_worker(client, auth, headless_env, monkeypatch):
+    monkeypatch.setenv("FAKE_CLAUDE_MODE", "hang")
+    _spawnable_project(headless_env["tmp"] / "proj")
+    client.post(
+        "/projects/proj/agents/spawn", json={"name": "api", "task": "hang"}, headers=auth
+    )
     worker.drain("w")
 
+    # The hanging run is live; kill flags it and the supervisor SIGTERMs its child.
     r = client.post("/projects/proj/agents/api/kill", headers=auth)
     assert r.status_code == 202
     worker.drain("w")
 
     assert client.get(f"/commands/{r.json()['id']}", headers=auth).json()["status"] == "done"
-    assert "proj__api" in fake_tmux["calls"]["kill_session"]
     with get_engine().begin() as conn:
-        assert repo.get_agent_by_name(conn, "proj", "api")["status"] == "done"
+        agent = repo.get_agent_by_name(conn, "proj", "api")
+    assert agent["status"] == "done"
+
+    def canceled():
+        with get_engine().begin() as conn:
+            run = repo.get_latest_run(conn, agent["id"])
+        return run if run["status"] != "running" else None
+
+    run = _wait(canceled, timeout=30.0)
+    assert run is not None, "kill never terminated the hanging run"
+    assert run["status"] == "canceled"

tests/test_reaper.py

@@ -0,0 +1,116 @@
+"""Worker heartbeat + reaper: a worker that stops heartbeating gets its running runs
+(and their still-'working' agents) marked crashed by any surviving worker — the positive
+liveness that replaces tmux scraping. No auto-requeue: half-done runs may have pushed."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime, timedelta
+
+from handler.control import worker
+from handler.db import repository as repo
+from handler.db.engine import get_engine
+
+
+def _seed_run(worker_id, agent_name="a", agent_status="working"):
+    with get_engine().begin() as conn:
+        if repo.get_project(conn, "p") is None:
+            repo.create_project(conn, "p", "/tmp/p")
+        agent = repo.create_agent(conn, "p", agent_name, f"/tmp/p/{agent_name}",
+                                  status=agent_status)
+        run = repo.create_run(conn, agent["id"], f"sid-{agent_name}", worker_id, "spawn")
+    return agent, run
+
+
+def _hb(worker_id, age_seconds=0.0):
+    with get_engine().begin() as conn:
+        repo.upsert_worker_heartbeat(conn, worker_id, hostname="h", pid=1)
+        if age_seconds:
+            from handler.db.tables import workers as workers_table
+
+            conn.execute(
+                workers_table.update()
+                .where(workers_table.c.id == worker_id)
+                .values(heartbeat_at=datetime.now(UTC) - timedelta(seconds=age_seconds))
+            )
+
+
+def test_reaper_marks_stale_workers_runs_crashed(env):
+    agent, run = _seed_run("w-dead")
+    _hb("w-dead", age_seconds=120)  # stale: default worker_stale_after is 60s
+
+    assert worker.reap_dead_workers() == 1
+
+    with get_engine().begin() as conn:
+        assert repo.get_run(conn, run["id"])["status"] == "crashed"
+        assert repo.get_agent_by_id(conn, agent["id"])["status"] == "crashed"
+        events = repo.list_agent_events(conn, agent["id"])
+        # The dead worker's registry row is dropped once settled.
+        assert repo.list_stale_workers(conn, datetime.now(UTC) + timedelta(days=1)) == []
+    assert events[0]["type"] == "worker"
+    assert events[0]["payload"]["worker_id"] == "w-dead"
+
+
+def test_reaper_leaves_fresh_workers_alone(env):
+    agent, run = _seed_run("w-live")
+    _hb("w-live")  # fresh heartbeat
+
+    assert worker.reap_dead_workers() == 0
+    with get_engine().begin() as conn:
+        assert repo.get_run(conn, run["id"])["status"] == "running"
+        assert repo.get_agent_by_id(conn, agent["id"])["status"] == "working"
+
+
+def test_reaper_preserves_paused_agent_status(env):
+    """paused_for_input still accurately describes what the agent needs — only agents
+    stuck in 'working' get flipped to crashed."""
+    agent, run = _seed_run("w-dead2", agent_name="paused", agent_status="paused_for_input")
+    _hb("w-dead2", age_seconds=120)
+
+    assert worker.reap_dead_workers() == 1
+    with get_engine().begin() as conn:
+        assert repo.get_run(conn, run["id"])["status"] == "crashed"
+        assert repo.get_agent_by_id(conn, agent["id"])["status"] == "paused_for_input"
+
+
+def test_reaper_idempotent_against_races(env):
+    agent, run = _seed_run("w-dead3", agent_name="raced")
+    _hb("w-dead3", age_seconds=120)
+    assert worker.reap_dead_workers() == 1
+    # A second reaper (or the same one next pass) finds nothing left to settle.
+    assert worker.reap_dead_workers() == 0
+    with get_engine().begin() as conn:
+        assert len(repo.list_agent_events(conn, agent["id"])) == 1
+
+
+def test_heartbeat_registers_worker(env):
+    worker.heartbeat("w-me")
+    with get_engine().begin() as conn:
+        rows = repo.list_stale_workers(conn, datetime.now(UTC) + timedelta(seconds=1))
+    assert [w["id"] for w in rows] == ["w-me"]
+    assert rows[0]["hostname"]
+    assert rows[0]["active_runs"] == 0
+
+
+def test_events_route_serves_stream(env, client, auth):
+    agent, run = _seed_run("w-api", agent_name="api-agent")
+    with get_engine().begin() as conn:
+        for seq in range(1, 4):
+            repo.insert_agent_event(
+                conn, agent["id"], run["id"], seq=seq, type="assistant",
+                payload={"n": seq}, session_id=run["session_id"],
+            )
+
+    resp = client.get("/projects/p/agents/api-agent/events", headers=auth)
+    assert resp.status_code == 200
+    events = resp.json()
+    assert [e["seq"] for e in events] == [1, 2, 3]
+    assert events[0]["type"] == "assistant"
+
+    # Cursor paging: only events after the given id come back.
+    resp = client.get(
+        f"/projects/p/agents/api-agent/events?after_id={events[1]['id']}", headers=auth
+    )
+    assert [e["seq"] for e in resp.json()] == [3]
+
+    resp = client.get("/projects/p/agents/missing/events", headers=auth)
+    assert resp.status_code == 404

tests/test_repository_runs.py

@@ -0,0 +1,174 @@
+"""Repository coverage for the headless-runner tables (workers / agent_runs /
+agent_events / session_archives) and the new claim filters. The ``env`` fixture runs the
+real ``alembic upgrade head``, so migration 0008 itself is under test here too."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime, timedelta
+
+from handler.db import repository as repo
+
+
+def _agent(conn, name="a1"):
+    repo.create_project(conn, "p", "/projects/p")
+    return repo.create_agent(conn, "p", name, "/projects/p")
+
+
+# ------------------------------------------------------------------------ agent_runs
+
+
+def test_run_lifecycle(conn):
+    agent = _agent(conn)
+    run = repo.create_run(conn, agent["id"], "sid-1", "worker-a", "spawn")
+    assert run["status"] == "running"
+    assert run["kind"] == "spawn"
+    assert run["cancel_requested"] is False
+
+    assert repo.finish_run(conn, run["id"], "completed", exit_code=0, result={"ok": True})
+    done = repo.get_run(conn, run["id"])
+    assert done["status"] == "completed"
+    assert done["exit_code"] == 0
+    assert done["result"] == {"ok": True}
+    assert done["finished_at"] is not None
+
+
+def test_finish_run_only_once(conn):
+    """The supervisor's verdict and a racing reaper can't clobber each other."""
+    agent = _agent(conn)
+    run = repo.create_run(conn, agent["id"], "sid", "w", "spawn")
+    assert repo.finish_run(conn, run["id"], "crashed") is True
+    assert repo.finish_run(conn, run["id"], "completed", exit_code=0) is False
+    assert repo.get_run(conn, run["id"])["status"] == "crashed"
+
+
+def test_cancel_request_roundtrip(conn):
+    agent = _agent(conn)
+    run = repo.create_run(conn, agent["id"], "sid", "w", "resume")
+    assert repo.get_cancel_requested(conn, run["id"]) is False
+    assert repo.request_run_cancel(conn, run["id"]) is True
+    assert repo.get_cancel_requested(conn, run["id"]) is True
+    # A finished run can't be re-flagged.
+    repo.finish_run(conn, run["id"], "canceled")
+    assert repo.request_run_cancel(conn, run["id"]) is False
+
+
+def test_list_running_runs_scoped_by_worker(conn):
+    agent = _agent(conn)
+    r1 = repo.create_run(conn, agent["id"], "s1", "worker-a", "spawn")
+    r2 = repo.create_run(conn, agent["id"], "s2", "worker-b", "spawn")
+    repo.finish_run(conn, r1["id"], "completed")
+    running = repo.list_running_runs(conn)
+    assert [r["id"] for r in running] == [r2["id"]]
+    assert repo.list_running_runs(conn, worker_id="worker-a") == []
+    assert [r["id"] for r in repo.list_running_runs(conn, worker_id="worker-b")] == [r2["id"]]
+
+
+def test_latest_run_and_agent_session(conn):
+    agent = _agent(conn)
+    repo.create_run(conn, agent["id"], "s1", "w", "spawn")
+    latest = repo.create_run(conn, agent["id"], "s1", "w", "resume")
+    assert repo.get_latest_run(conn, agent["id"])["id"] == latest["id"]
+
+    repo.set_agent_session(conn, agent["id"], "s1", "worker-a")
+    updated = repo.get_agent_by_id(conn, agent["id"])
+    assert updated["session_id"] == "s1"
+    assert updated["worker_id"] == "worker-a"
+
+
+def test_agent_status_crashed_allowed(conn):
+    """Migration 0008 widened ck_agents_status — 'crashed' must insert cleanly."""
+    agent = _agent(conn)
+    repo.set_agent_status(conn, agent["id"], "crashed")
+    assert repo.get_agent_by_id(conn, agent["id"])["status"] == "crashed"
+
+
+# ---------------------------------------------------------------------- agent_events
+
+
+def test_events_cursor_pagination(conn):
+    agent = _agent(conn)
+    run = repo.create_run(conn, agent["id"], "sid", "w", "spawn")
+    for seq in range(1, 4):
+        repo.insert_agent_event(
+            conn, agent["id"], run["id"], seq=seq, type="assistant",
+            payload={"n": seq}, session_id="sid",
+        )
+    first = repo.list_agent_events(conn, agent["id"], limit=2)
+    assert [e["payload"]["n"] for e in first] == [1, 2]
+    rest = repo.list_agent_events(conn, agent["id"], after_id=first[-1]["id"])
+    assert [e["payload"]["n"] for e in rest] == [3]
+    assert repo.list_agent_events(conn, agent["id"], after_id=rest[-1]["id"]) == []
+
+
+# ------------------------------------------------------------------ session_archives
+
+
+def test_session_archive_upsert_replaces(conn):
+    agent = _agent(conn)
+    repo.upsert_session_archive(conn, agent["id"], "s1", b"v1")
+    repo.upsert_session_archive(conn, agent["id"], "s2", b"v2-longer")
+    row = repo.get_session_archive(conn, agent["id"])
+    assert row["session_id"] == "s2"
+    assert bytes(row["archive"]) == b"v2-longer"
+    assert row["bytes"] == len(b"v2-longer")
+
+
+def test_session_archive_missing(conn):
+    agent = _agent(conn)
+    assert repo.get_session_archive(conn, agent["id"]) is None
+
+
+# ------------------------------------------------------------------------- workers
+
+
+def test_worker_heartbeat_upsert_and_staleness(conn):
+    repo.upsert_worker_heartbeat(conn, "worker-a", hostname="h1", pid=42, max_runs=4)
+    repo.upsert_worker_heartbeat(conn, "worker-a", hostname="h1", pid=42, active_runs=2)
+
+    future = datetime.now(UTC) + timedelta(seconds=1)
+    stale = repo.list_stale_workers(conn, cutoff=future)
+    assert [w["id"] for w in stale] == ["worker-a"]
+    assert stale[0]["active_runs"] == 2
+
+    past = datetime.now(UTC) - timedelta(minutes=5)
+    assert repo.list_stale_workers(conn, cutoff=past) == []
+
+
+# ------------------------------------------------------------------- claim filters
+
+
+def test_claim_respects_target_worker(conn):
+    repo.create_project(conn, "p", "/projects/p")
+    pinned = repo.enqueue_command(conn, "login_submit", payload={"code": "x"},
+                                  target_worker="worker-b")
+    # worker-a can't see worker-b's pinned command...
+    assert repo.claim_next_command(conn, "worker-a") is None
+    # ...but worker-b claims it.
+    claimed = repo.claim_next_command(conn, "worker-b")
+    assert claimed["id"] == pinned["id"]
+    assert claimed["status"] == "running"
+
+
+def test_claim_excludes_types_when_slots_full(conn):
+    repo.create_project(conn, "p", "/projects/p")
+    spawn_cmd = repo.enqueue_command(conn, "spawn", project_id="p", agent_name="a")
+    sync_cmd = repo.enqueue_command(conn, "sync", project_id="p")
+    # With run-starting types excluded, the older spawn is skipped for the sync.
+    claimed = repo.claim_next_command(
+        conn, "w", types_excluded=("spawn", "resume", "mise_init")
+    )
+    assert claimed["id"] == sync_cmd["id"]
+    # The spawn stays queued for a worker with a free slot.
+    assert repo.claim_next_command(conn, "w2")["id"] == spawn_cmd["id"]
+
+
+def test_delete_agent_cascades_headless_rows(conn):
+    agent = _agent(conn)
+    run = repo.create_run(conn, agent["id"], "sid", "w", "spawn")
+    repo.insert_agent_event(conn, agent["id"], run["id"], seq=1, type="system", payload={})
+    repo.upsert_session_archive(conn, agent["id"], "sid", b"data")
+
+    assert repo.delete_agent(conn, "p", agent["name"]) is True
+    assert repo.get_latest_run(conn, agent["id"]) is None
+    assert repo.list_agent_events(conn, agent["id"]) == []
+    assert repo.get_session_archive(conn, agent["id"]) is None

tests/test_worker.py

@@ -65,8 +65,8 @@ def test_resume_command_feeds_answer_and_sets_working(env, monkeypatch):
     _seed_project("api")
     seen = {}
 
-    def fake_resume(agent, ans):
+    def fake_resume(agent, ans, worker_id=None):
-        seen.update(name=agent["name"], ans=ans)
+        seen.update(name=agent["name"], ans=ans, worker_id=worker_id)
         return True, "ok"
 
     monkeypatch.setattr(spawn, "resume", fake_resume)
@@ -74,11 +74,29 @@ def test_resume_command_feeds_answer_and_sets_working(env, monkeypatch):
 
     worker.drain("w")
     assert _get(cmd["id"])["status"] == "done"
-    assert seen == {"name": "api", "ans": "Postgres"}
+    assert seen == {"name": "api", "ans": "Postgres", "worker_id": "w"}
     with get_engine().begin() as conn:
         assert repo.get_agent_by_name(conn, "p", "api")["status"] == "working"
 
 
+def test_resume_command_fails_loudly_when_undeliverable(env, monkeypatch):
+    """An undeliverable answer must surface as a FAILED command — never a silent 'done'
+    (the original bug: send-keys into a dead tmux pane reported success)."""
+    _seed_project("api")
+    monkeypatch.setattr(
+        spawn, "resume", lambda agent, ans, worker_id=None: (False, "no live session")
+    )
+    cmd = _enqueue(type="resume", project_id="p", agent_name="api", payload={"answer": "x"})
+
+    worker.drain("w")
+    failed = _get(cmd["id"])
+    assert failed["status"] == "failed"
+    assert "no live session" in failed["error"]
+    with get_engine().begin() as conn:
+        # The agent must NOT be flipped to working when nothing was delivered.
+        assert repo.get_agent_by_name(conn, "p", "api")["status"] != "working"
+
+
 def test_approve_command_records_operator_verdict_with_head_sha(env, fake_gitops):
     _seed_project("senior")
     cmd = _enqueue(
@@ -211,45 +229,6 @@ def test_bad_command_is_recorded_failed_not_raised(env):
     assert "agent name" in failed["error"]
 
 
-def test_capture_agent_output_snapshots_working_agents(env, monkeypatch):
-    with get_engine().begin() as conn:
-        repo.create_project(conn, "p", "/tmp/p")
-        agent = repo.create_agent(conn, "p", "api", "/tmp/p/api", status="working")
-
-    monkeypatch.setattr(worker.tmux, "has_session", lambda name: True)
-    monkeypatch.setattr(
-        worker.tmux, "capture_pane", lambda name, escapes=False: "boot\nTheme picker\n\n\n"
-    )
-
-    assert worker.capture_agent_output() == 1
-    with get_engine().begin() as conn:
-        row = repo.get_agent_by_id(conn, agent["id"])
-    # The tail is stored with trailing blank lines trimmed.
-    assert row["last_output"] == "boot\nTheme picker"
-    assert row["output_at"] is not None
-
-
-def test_capture_agent_output_skips_dead_sessions_and_nonworking(env, monkeypatch):
-    with get_engine().begin() as conn:
-        repo.create_project(conn, "p", "/tmp/p")
-        repo.create_agent(conn, "p", "gone", "/tmp/p/gone", status="working")
-        done = repo.create_agent(conn, "p", "done", "/tmp/p/done", status="done")
-
-    captured = []
-    monkeypatch.setattr(worker.tmux, "has_session", lambda name: False)
-    monkeypatch.setattr(
-        worker.tmux,
-        "capture_pane",
-        lambda name, escapes=False: captured.append(name) or "x",
-    )
-
-    # The working agent's session is dead (skipped); the done agent isn't queried at all.
-    assert worker.capture_agent_output() == 0
-    assert captured == []
-    with get_engine().begin() as conn:
-        assert repo.get_agent_by_id(conn, done["id"])["last_output"] is None
-
-
 def test_drain_processes_multiple_then_stops(env, monkeypatch):
     _seed_project()
     monkeypatch.setattr(poller, "sweep", lambda project_id=None: {"checked": 0})

tests/test_worker_concurrency.py

@@ -0,0 +1,76 @@
+"""Slot-aware command claiming: a worker at max_concurrent_runs must leave run-starting
+commands queued (for a less-loaded worker) while still processing everything else. Slot
+accounting is DB-driven — this worker's ``running`` agent_runs rows — so it needs no
+in-memory registry and is exercised here without real subprocesses."""
+
+from __future__ import annotations
+
+import pytest
+
+from handler.control import spawn, worker
+from handler.db import repository as repo
+from handler.db.engine import get_engine
+
+
+@pytest.fixture
+def headless_env(env, monkeypatch):
+    from handler import config
+
+    monkeypatch.setenv("MAX_CONCURRENT_RUNS", "2")
+    config.get_settings.cache_clear()
+    yield env
+    config.get_settings.cache_clear()
+
+
+def _seed(conn_count_running_for=None):
+    with get_engine().begin() as conn:
+        repo.create_project(conn, "p", "/tmp/p")
+        agent = repo.create_agent(conn, "p", "a", "/tmp/p/a")
+        return agent
+
+
+def _running_run(agent_id, worker_id):
+    with get_engine().begin() as conn:
+        return repo.create_run(conn, agent_id, f"sid-{worker_id}", worker_id, "spawn")
+
+
+def test_full_worker_skips_run_commands_but_processes_others(headless_env, monkeypatch):
+    agent = _seed()
+    _running_run(agent["id"], "w-full")
+    _running_run(agent["id"], "w-full")  # 2 running == MAX_CONCURRENT_RUNS
+
+    spawned = {}
+    monkeypatch.setattr(
+        spawn, "spawn",
+        lambda project_id, name, **kw: spawned.update(name=name, **kw)
+        or {"id": 1, "name": name, "working_dir": "/tmp/p/x", "forge_note": None},
+    )
+    with get_engine().begin() as conn:
+        spawn_cmd = repo.enqueue_command(conn, "spawn", project_id="p", agent_name="x")
+        kill_cmd = repo.enqueue_command(conn, "kill", project_id="p", agent_name="a")
+    monkeypatch.setattr(spawn, "kill", lambda p, n: None)
+
+    # The full worker processes the kill but leaves the spawn queued.
+    assert worker.drain("w-full") == 1
+    with get_engine().begin() as conn:
+        assert repo.get_command(conn, kill_cmd["id"])["status"] == "done"
+        assert repo.get_command(conn, spawn_cmd["id"])["status"] == "queued"
+    assert spawned == {}
+
+    # A worker with free slots picks the spawn up.
+    assert worker.drain("w-free") == 1
+    with get_engine().begin() as conn:
+        assert repo.get_command(conn, spawn_cmd["id"])["status"] == "done"
+    assert spawned["name"] == "x"
+    assert spawned["worker_id"] == "w-free"
+
+
+def test_slot_frees_when_run_finishes(headless_env, monkeypatch):
+    agent = _seed()
+    run1 = _running_run(agent["id"], "w1")
+    _running_run(agent["id"], "w1")
+    assert worker._full_slot_exclusions("w1") == worker._RUN_COMMANDS
+
+    with get_engine().begin() as conn:
+        repo.finish_run(conn, run1["id"], "completed", exit_code=0)
+    assert worker._full_slot_exclusions("w1") == ()