Choose the LLM route by model, and add a capability matrix #34

2026-09-22T23:32:47Z

dev-agent commented

2026-09-22 23:32:47 +00:00

The new models made a latent bug observable and gave us something to measure.

The bug

The delegate's client was hardcoded to the Anthropic passthrough. Pointing NISSE_ROLE_DELEGATE at gpt-6-astra or a local model would have sent Anthropic-shaped /v1/messages bodies to something that does not speak them. A5 promises the frontier tier is swappable "as a configuration change, not a refactor", and it simply was not — the promise held only while every configured model happened to be Anthropic.

select_client now picks the route from the model: the vendor-native adapter where one exists, LiteLLM's normalization otherwise. That is precisely what A13 prescribes for a vendor whose renderer adapter has not been written — provisionally accept normalization, and validate it against the A16 eval set before trusting delegations to it.

The capability matrix

Part H says the thresholds that depend on local capability — A12's failure count, A14's consult budget, A15's manifest size — must be tuned "at build-order step 5 on the real hardware". That needs measurement, not reputation.

The harness probes each model for the things that actually break, all of which we have hit for real:

does it call a tool at all, or answer in prose (the planner bug, twice)
does the plan it submits have substance — llama3.2 emitted { as an entire plan, which is structurally a valid tool call and substantively nothing
can it consume its own tool result, which the executor loop depends on
does prompt-level JSON hold, since A14 contracts structured output there rather than at the API level
does its selected route actually work, which is the regression test for the bug above

Opt in with NISSE_CAPABILITY_MODELS; it spends real tokens across several models, so it stays out of ordinary runs.

Note on keys

The two keys turn out to be complementary rather than overlapping — the test key reaches claude-haiku-4-5 and llama3.2, the production key reaches claude-opus-5, gpt-6-astra, local-coder and local-fast. So a full six-model matrix needs two passes today. Unit tests continue to use the test key unchanged.

279 unit + 17 integration green locally.

🤖 Generated with Claude Code

The new models made a latent bug observable and gave us something to measure. ### The bug The delegate's client was **hardcoded to the Anthropic passthrough**. Pointing `NISSE_ROLE_DELEGATE` at `gpt-6-astra` or a local model would have sent Anthropic-shaped `/v1/messages` bodies to something that does not speak them. A5 promises the frontier tier is swappable "as a configuration change, not a refactor", and it simply was not — the promise held only while every configured model happened to be Anthropic. `select_client` now picks the route from the model: the vendor-native adapter where one exists, LiteLLM's normalization otherwise. That is precisely what A13 prescribes for a vendor whose renderer adapter has not been written — provisionally accept normalization, and validate it against the A16 eval set before trusting delegations to it. ### The capability matrix Part H says the thresholds that depend on local capability — A12's failure count, A14's consult budget, A15's manifest size — must be tuned "at build-order step 5 on the real hardware". That needs measurement, not reputation. The harness probes each model for the things that actually break, all of which we have hit for real: - does it call a tool at all, or answer in prose (the planner bug, twice) - does the plan it submits have substance — llama3.2 emitted `{` as an entire plan, which is structurally a valid tool call and substantively nothing - can it consume its own tool result, which the executor loop depends on - does prompt-level JSON hold, since A14 contracts structured output there rather than at the API level - does its selected route actually work, which is the regression test for the bug above Opt in with `NISSE_CAPABILITY_MODELS`; it spends real tokens across several models, so it stays out of ordinary runs. ### Note on keys The two keys turn out to be complementary rather than overlapping — the test key reaches `claude-haiku-4-5` and `llama3.2`, the production key reaches `claude-opus-5`, `gpt-6-astra`, `local-coder` and `local-fast`. So a full six-model matrix needs two passes today. Unit tests continue to use the test key unchanged. 279 unit + 17 integration green locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-22 23:32:47 +00:00

Choose the LLM route by model, and add a capability matrix

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 58s

Details

ed2f5dfc7f

The arrival of non-Anthropic models made a latent bug observable: the
delegate's client was hardcoded to the Anthropic passthrough, so
pointing NISSE_ROLE_DELEGATE at gpt-6-astra or a local model would
have sent Anthropic-shaped /v1/messages bodies to something that does
not speak them. A5 promises the frontier tier is swappable "as a
configuration change, not a refactor", and it was not.

select_client now picks the route from the model: the vendor-native
adapter where one exists, LiteLLM's normalization otherwise — which is
exactly what A13 prescribes for a vendor whose renderer adapter has
not been written, pending validation against the A16 eval set.

Also adds the capability matrix Part H asks for. It says the
thresholds that depend on local capability — A12's failure count,
A14's consult budget, A15's manifest size — must be tuned "on the real
hardware", and that needs measurement rather than reputation. The
harness probes each model for the things that actually break: whether
it calls a tool at all, whether the plan it submits has substance
(llama3.2 emitted "{"), whether it can consume its own tool result,
and whether prompt-level JSON holds. Opt in with
NISSE_CAPABILITY_MODELS; it spends real tokens across several models.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-22 23:45:30 +00:00

Detect truncated responses, and raise the output ceiling

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 47s

Details

249bcb10fc

Measuring Opus found a bug that had been silent all along: nothing
checked stop_reason or finish_reason, so a response cut off mid-answer
was parsed as a complete one.

The symptom was a submit_plan tool call whose required `content` field
was simply absent — structurally valid, substantively empty, and
indistinguishable from a model that chose to send nothing. The cause
was our max_tokens of 4096: Opus writing an architecture plan
routinely spends 3,000-4,000 output tokens, and was observed at 4,064
against that ceiling. Roughly one run in three crossed it.

Worse than the empty plan is what the same failure does elsewhere: a
truncated write_file writes half a file and reports success.

- Both routes now raise TruncatedResponse when the model was cut off.
  It subclasses LLMError, so the planner's retry and the executor's
  and delegation's gating already handle it without learning a new
  type: a cut-off turn is a failed turn
- The default ceiling is 16384 and configurable via
  NISSE_MAX_OUTPUT_TOKENS. Output is billed by what is produced, not
  by the ceiling, so a low limit buys nothing and costs silent
  truncation

Verified: three consecutive Opus planning turns now return plans of
5,700-6,200 characters where one in three previously returned nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-22 23:54:58 +00:00

Enforce the token cap, and treat a spent budget as an interruption

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 51s

Details

8cfb45aed3

Exceeding a LiteLLM spend limit takes external models away for the
rest of the day, which makes three existing gaps consequential rather
than untidy.

- A13 calls the delegation caps hard on "turns, tokens, and wall
  clock", and the module docstring repeated it, but the token cap was
  decoration: nothing counted, and result.tokens was always zero. It
  is now metered from the usage the provider reports, so a confused
  delegate cannot spend a day's allowance. A5 keeps the *money* cap on
  the LiteLLM key, which is a different thing; this is a scope cap
- The normalized route sent no ceiling at all, so truncation detection
  there had nothing to detect against. Both routes now send one
- A budget refusal is recognised distinctly. It is not a rate limit:
  retrying cannot help, so the planner no longer burns a second call
  to fail identically

And a delegation cut off by an exhausted budget is now "interrupted"
rather than "failed", which preserves its branch and leaves its intent
pending. The work on that branch may be perfectly good; discarding it
because the budget ran out would throw away work for a reason that has
nothing to do with the work. A3's resume rule already knows how to
pick such a delegation back up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-22 23:57:07 +00:00

Preserve delegation work through any upstream outage, not just budget

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 52s

Details

8cfa116aa1

An exhausted budget was a special case of something more general: the
work is not at fault. A network blip, a timeout, a 5xx from the proxy,
or a rate limit that did not clear all leave a delegation branch that
may be perfectly good, and discarding it would throw away work for a
reason that has nothing to do with the work.

UpstreamUnavailable now names that family, with BudgetExhausted a
member of it — distinguishable where the difference matters, since a
budget will not clear within the hour while a network blip will. A
delegation stopped by any of them is "interrupted": branch kept,
intent left pending, and A3's resume rule picks it up.

The benefit of the doubt stops at the server's own door. A 4xx that is
not a rate limit means the request itself was wrong, and preserving
branches on the strength of one would hide a real bug — so those still
fail and clean up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dongar approved these changes

2026-09-23 01:47:26 +00:00

dongar merged commit 231b406393 into main

2026-09-23 01:47:35 +00:00

dongar deleted branch model-routing

2026-09-23 01:47:35 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#34