Delivery report: the session router (v0.5.0)

Status: implemented and verified on local main. This report covers the per-session router selector, the investigation and repair of NobodyWho's routing answers, and the optional JEV provider. It supersedes section 4 of delivery-v0.5.0-hardening.md: the localAdvice option described there never shipped and is removed.

What was built#

A session runs in one of three mutually exclusive router modes, chosen by the person with /cockpit router and held in session state:

/cockpit router with no word shows the mode and, in an interactive session, offers the three in the engine's own dialog. /cockpit router off|nobodywho|jev switches directly. Selecting a provider sends one fixed availability question that carries nothing of the session, so the state shown (NOBODYWHO · LOCAL, JEV · CONNECTED, JEV · UNAVAILABLE (jev_disabled)) is observed, not assumed. /cockpit and /cockpit version show the mode. Nothing is written to settings, to the plugin store's preferences or to any router configuration.

Code: hooks/router.ts (pure: modes, the question, what counts as an answer, the adjudication, the records) and three functions in hooks/register.tsx (askRouter, the only place a router process starts; routeTask; selectRouter).

NobodyWho: root cause#

The earlier evaluation found that the local classifier chose the first-listed option 87.5% of the time and looked stable while doing it. The cause is in two places.

1. The models have a strong prior that is unrelated to the task. The router's tier-1 specialist (a 0.6B fine-tuned option-token classifier) follows position on a two-way question, and on a five-way question answers one label ("delegate") for 52 of 56 tasks whatever the order.
2. The router's own debiasing does not operate on short lists. The provider shuffles the options per sample from fixed seeds (1234, 1235, 1236) and stops early when two samples agree. For a two-option question all three seeds produce the identity permutation, so every sample sees the caller's order, the first two agree, and sampling stops. The reported sample_stability of 0.67 to 1.0 is one biased draw counted twice. For three options, two of the three seeds are again identical.

The router is the operator's own infrastructure and is not changed here. The correction is on Cockpit's side of the interface.

The corrected, position-independent format#

A router is never shown a list it can pick a position from and be believed. Instead:

• Fixed features, not routes. Five yes/no questions about the task: security-sensitive, architectural, hard debugging, needs a broad search, has independent parts. The route follows from the features by a fixed priority (routeOfFeatures), in Cockpit, not in the model.
• Both orders, always. Each question is asked yes-first and no-first: ten requests, issued at the same time. A feature counts only when its two answers agree.
• A validity floor. If fewer than four of the five features agree across orders, the reading is position, not judgement, and is discarded whole.
• A cap. A reading that affirms more than three features is an answer to everything and is discarded. No measured reading from a working provider affirmed more than three.
• Provider check. An answer counts only with a receipt and only from the provider the mode names. NobodyWho mode never accepts a JEV answer, or the reverse.

No stability is manufactured by repeating one order: a provider that follows position fails the floor and deterministic policy decides.

Evaluation#

56 synthetic tasks, eight for each of seven kinds: trivial, ordinary, difficult, architectural, security-sensitive, discovery and independent-parts. The expected routes are direct, direct, effort, Opus, Opus, scout and delegate. In each kind four prompts use plain vocabulary and four are paraphrased to avoid obvious keywords. The cases were written before the lexical rules for the three new features. Every provider saw the same cases in the same request shape. Calls: 896 each for the two local models (including a five-rotation vote that helped neither and was dropped), 672 for JEV. No call failed.

The corrected-format rows are the recorded answers replayed through the shipped readAnswers.

Confusion, corrected format (rows: task kind; columns: route read):

Conclusions the data supports:

• The corrected format is a measured improvement for the plain local model: 43 to 49 correct, with position sensitivity removed by construction.
• The tier-1 specialist is not a usable router for this question in any format. NobodyWho mode therefore uses the router's plain local provider (--mode local), which is also what guarantees that mode never reaches JEV. Two thirds of the specialist's readings (37 of 56) fail the validity checks in any case.
• JEV is the more accurate reader, by a small margin on this set (51 against 49).
• Both useful providers are better than the lexical rules on paraphrased prompts (24 and 25 of 28 against 17) and no better on plain ones (25 and 26 against 28).

What it does not support: these are 56 synthetic prompts written by the author of the rules, on one machine. They do not establish accuracy on real work.

As shipped: the rules go first#

Cockpit's rules remain authoritative, so the router's reading is not the outcome. Replaying the same answers through the shipped routeTask logic and adjudicate:

The automatic gain is small, and deliberately so. Most of what the routers add is "this is architectural or security-sensitive", and a router is not allowed to make an Opus consultation happen. Those readings are refused (6 for Qwen3 4B, 8 for JEV, every one of them a task that really was architectural or security-sensitive) and shown to the person as a notice. Saying "ask Opus" is then an explicit request, a ground the rules already honour. JEV's other additions were 5 routes, 3 correct; the two wrong ones recommended delegation for an ordinary task, which is advice the main session may ignore. The unreliable specialist changed no outcome.

JEV#

Where it was switched off, and why it still is. Two places on the operator's machine, neither changed:

1. The decision router's own configuration, jev.enabled: false: a hard switch. With it false the router never calls JEV in any mode. It is also the router's shipped default.
2. The shell wrappers that start agent sessions unset the TypeSafe key and endpoint variables, so no session or child process inherits them. This does not block the router, which reads its key from its own key file.

How JEV mode reaches it. The router has no per-request way to enable JEV. It does read its configuration directory from DECISION_ROUTER_CONFIG_DIR. The operator creates one directory holding a copy of the router configuration with JEV enabled and names it in the new routerConfigDir option. Cockpit passes that variable to the router's child process, and only for requests made while the session's router is JEV. Cockpit never creates, reads or edits router configuration, and the key never enters the plugin's process. Without routerConfigDir, JEV mode runs under the router's own switch and reports UNAVAILABLE (jev_disabled) when that is off. The global switch stays off, so no other caller on the machine can reach JEV.

Measured. 672 evaluation requests and 50 live ones (two rounds of 25) through the approved configuration, all answered by jev-1.13.0, with no error and no timeout. A further 22 live requests without that configuration were refused by the router's own switch, as intended.

The 16 ms figure sometimes quoted for JEV was not observed: a request took about fifteen times that from this machine. Each request reported about 380 input tokens and 40 output tokens of JEV usage; a reading is ten.

Privacy boundary. JEV is an external service. In JEV mode the request carries the first 400 characters of the prompt that starts a task and one fixed question. It carries no file content, no path Cockpit observed and no history. A prompt in which credential-shaped text is recognised anywhere is not sent at all, and is recorded as a fallback with no receipt. The filter is a broad list of shapes (named values, URLs carrying a password, known token prefixes, .env lines, long unbroken runs, "the password is …"), not a guarantee. Outside a planned task every prompt is a task of its own, so in JEV mode each such prompt's first 400 characters are sent. The selection reply states this boundary each time JEV is chosen.

Authority boundaries#

A router's recommendation is judged by adjudicate against the rules. Accepting one starts nothing.

It is asked once per task, at its start, when the rules do not settle it. A plan in progress is one task, and prompts inside it are not asked about. Without a plan, each prompt of 16 characters or more is its own task and is asked about. It is never asked for a tool call, a helper, or an Opus admission.

Evidence#

Each decision is one bounded record in the Run Ledger (12 SESSION ROUTER, the last sixteen) and in session state: the mode, the provider and model that really answered, up to four receipt ids, requests sent and answered, wall time by the session's clock and the provider's own latencies summed, how many features agreed, the recommendation, what the rules said, and whether the rules accepted it, refused it or it was discarded, with the reason. No prompt text is kept. No receipt is recorded for a request that did not run. Router figures are a separate section and are never counted as model requests or tokens.

Switching#

Tested in every direction (OFF to NobodyWho, NobodyWho to JEV, JEV to OFF, OFF to JEV, JEV to NobodyWho), in host-hook tests and in one live session. Each switch raises an epoch and clears the previous provider's tallies and last decision. A reading still in flight when the person switches is dropped: not recorded, not said. The main model and its effort are unaffected in both profiles.

Live acceptance (isolated)#

Run on the final code, after the review fixes. Claude Code 2.1.296, one Cockpit per session (the working tree, loaded inline, with the installed copy disabled for that process only), cobaltStrict: true, profile: SONNET_LED, Sonnet at medium. Router receipts are attributed by the session's working directory in the router's own ledger.

Overhead#

• OFF: none. No process, no request, no added text.
• Context: no tool schema is added. One sentence of the system prompt changed (about 40 tokens net). An accepted recommendation adds one line of about 45 tokens to the prompt that starts the task.
• Time, per task the rules do not settle: about 1.1 s with NobodyWho (ten concurrent local requests; about 2.0 s if issued one after another), about 0.4 to 0.5 s with JEV, and never more than the per-request timeout (5 s local, 3 s JEV). Outside a plan that is per prompt. Selecting a provider costs one request (0.3 to 0.5 s; 1.85 s when the local model had to load).
• JEV usage: ten small requests per reading.

Tests#

tests/router.test.ts: 48 tests. Pure: modes and parsing; the baseline routes; what a router is shown, including clipping, non-tasks and credential withholding; the ten requests; answer parsing; position-following, yes-to-everything, wrong-provider, no-receipt and failed readings; every adjudication rule; records, labels and the ledger section. Through the real host hooks with a scripted router: OFF asks nothing; each mode's argument array and environment; the configuration directory reaching JEV calls only; the router's own refusal; the rules settling a task without a request; a router unable to cause Opus, lift a hold or start a helper; unreliable providers discarded; a credential prompt withheld; all switches; a reading dropped after a switch; the main model and effort unmoved; no re-routing inside a task; the dialog; the pane; nothing stored.

Nine mutations each fail at least one test: the configuration directory given to non-JEV calls; the stale-reading check removed; Opus accepted without a ground; the rules-first short-circuit removed; the agreement floor at zero; the credential withhold removed; a configuration directory inside the project accepted; a non-person command allowed to switch; the reset at session end removed.

Independent review#

Two independent Sonnet reviewers read the candidate before release: one the router commit, one the complete v0.5.0 change set against v0.4.0. Neither found an authority bypass or an injection path in the router: its arguments are fixed words and one JSON body, every field read from the router's output is whitelisted, the advisory line is fixed text, and nothing connects a recommendation to admission, the review hold, a model, an effort, ownership or a spawn.

Findings on the router, all fixed:

Findings on the rest of v0.5.0 that this release also fixes (details in the changelog): a mandatory Opus review could be discharged without an answer from Opus in four ways; the architecture ground could be raised by the main session's own plan length; and the Glob pattern refusal applied to helpers that own everything, which 0.4.0 allowed. Each has a test that fails when its fix is removed.

Left as it is, and why: consult admission is check-then-act across awaits, so two simultaneous requests could both be admitted, but the spawn budget still allows one running Opus. A file literally named with a leading ~ now counts as a wildcard. Whether an edit made through a shell command to a security-sensitive file raises the security ground depends on the shell-diff reader, which this release does not change.

Limits#

• The evaluation is small and synthetic, and the automatic gain over the rules is one or two tasks in 56.
• NobodyWho's quality is the quality of the router's local model. With a weak one most readings are discarded and the rules decide, which is safe but adds a second of latency for nothing.
• A reading costs ten requests. A two-request route question was as accurate for JEV, but the specialist passes its order check while being wrong, so it is not used.
• The router is consulted only at the start of a task. It does not reconsider as a task grows.
• In a headless single-prompt run there is no earlier message in which to select a router, so the mode is OFF unless several messages share the session.
• JEV sends a prompt excerpt to a third party. That is the operator's decision each session, and Cockpit says so each time.