13 Commits

Include renames

Author	SHA1	Message	Date
dongarandClaude Opus 5.5	50b46821d7	Add an executor benchmark that keeps its results ... Choosing a local executor model, and tuning its knobs, needs measured success rates across repeated runs, comparable with every model measured before. - Four canned milestones under nisse_dev/bench/tasks: write a function, fix a bug a failing test exposes, thread an option through several files, and find one failure in output far past max_output_chars. Each is an approved plan over a seed project, judged by a hidden check the model never sees; a reference solution proves each is solvable and that its check tells the difference - Runs use the agent's own client wiring, now shared as make_llm_clients, with delegation and ask_expert off: an escalation is a failure of the model under test - Each run is stored as one JSON line in bench/results/, committed, with the model's backing name and declared facts, a note on how it was served, the task's content hash, the agent's commit and tuning, the outcome, turns, misbehaviour counts, context trims, tokens, time and LiteLLM's cost; a fallback answer can never count as a success - The report rebuilds the comparison from every stored run, with Wilson 95% intervals, marking runs against older task versions - scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes a new model through the context probe, the capability matrix and the benchmark in one command - Clients keep LiteLLM's per-call cost, and BudgetedClient counts the requests it had to trim - The tuning guide gains "Benchmarking models" Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-25 00:35:24 +00:00
dongarandClaude Opus 5.5	b934c46714	Enforce the tool-call contract, asking servers only where it works ... The executor ended a milestone on any reply without a tool call, and nothing held a model to the offered tools. Measured on the 4070: tool_choice "required" is enforced by vLLM (local-fast) and Anthropic, but qwen3-14b (local-coder) under it cannot stop and writes until the output ceiling, a known model bug. So enforcement is asked for per model, and the loop holds the contract itself everywhere. - The catalog reads supports_tool_choice from LiteLLM's model_info; BudgetedClient sends tool_choice only to models declared true, as "required" on the normalized route and {"type": "any"} on the Anthropic passthrough, and only when tools are offered - The executor gains an explicit finish tool, the only way to end a milestone. Finishing with nothing changed is refused once; a prose reply gets one nudge, then escalates; a call to a tool not offered counts toward failure_limit, like a failed command; calls after finish are recorded as not run - Clients record the model that actually answered. ModelFacts.answered recognises a proxy fallback, which once credited Claude's answers to a local model that could not call tools at all - The context probe reports a fallback instead of trusting it; rerun, both local models now refuse over-long prompts, as they should - The capability matrix checks every answer came from the model asked, that models call only offered tools, and that a declared supports_tool_choice is true - Spec A15 and the tuning guide say how enforcement is requested, declared and backed up Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-24 22:54:49 +00:00
dongarandClaude Opus 5.5	494715efa9	Fit every model request to its context window ... Every turn re-sent the whole transcript with nothing checking it against the model's window. On a local model that fails silently: Ollama drops the start of an over-long prompt, which is where the system prompt and the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with no error. - The model catalog reads max_input_tokens and max_output_tokens from LiteLLM's model_info, and startup refuses a role on a model with no declared window - BudgetedClient wraps both routes. It fits the canonical events before rendering: older tool results cut to head and tail, then the oldest turns dropped with a note, keeping calls with their results; a request that still cannot fit fails with ContextOverflow and is never sent. The output ceiling is the smaller of config and the model's limit - Tokens are estimated from characters, pessimistically, and the estimate is corrected from reported prompt sizes, only ever upward - The executor takes its scope from the log: the latest plan, which is the approved milestone plan, goes in its system prompt, and it sees only events after it - Tool output keeps its head and, mostly, its tail, so a test runner's failure summary survives truncation - tuning.context.keep_recent_turns and safety_margin are new knobs - scripts/probe_context.py checks a declared window against what the server does; it confirmed llama3.2 cuts at the declared 4096 - The tuning guide covers the budget, declaring a local model's limits, and the probe Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-24 00:26:30 +00:00
dongarandClaude Opus 5.5	15d69799bf	Make the config file the only home for settings ... With no production instance yet, there is no reason to carry two ways of configuring the agent. Every environment variable that duplicated a config key is gone; the environment now supplies only the secrets the file references by ${…}, plus NISSE_CONFIG and NISSE_ENV to find the files. - AgentConfig loads from a file (load, from_file) or a parsed tree (from_tree, for tests); a missing file is an error naming the example - The from_env constructors on LiteLLMClient, AnthropicPassthroughClient, GiteaClient, MatrixConfig and RoleMap are removed, with the env reading for max output tokens; clients take their settings from config - The live integration tests share a committed agent config, tests/integration/nisse-test.yaml, aimed at test resources, and one conftest that loads it and builds clients. NISSE_LIVE_PLANNER_MODEL becomes its planner role; NISSE_CAPABILITY_MODELS stays as a per-run switch - A missing secret now fails the live tests instead of skipping them - CI passes only the three tokens - .env.example is secrets only; spec A5 says config comes from the file Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-23 23:25:58 +00:00
dongarandClaude Opus 5	e38adf40ca	Discover a model's vendor from LiteLLM instead of its name ... Route selection sniffed model names: anything starting "claude-" or "anthropic/" took the vendor-native passthrough, everything else took the normalized route. That works only while an alias names its vendor. A semantic role alias like "remote-architecture" is meant to point at Anthropic today and somewhere else tomorrow, and the name says nothing either way -- so it would have taken the normalized route regardless of what was behind it, silently losing the fidelity A13 wants for delegations. LiteLLM already knows the answer. /model/info reports each alias's backing model, provider, and declared tool support, so the agent asks at startup and routes on the provider it discovers. That keeps vendor knowledge where A5 already put it -- in the proxy -- rather than duplicating it as name patterns here. There is deliberately no name-based fallback: PASSTHROUGH_PREFIXES and has_native_adapter are gone, and select_client now requires a catalog. A second heuristic could only ever disagree with the proxy. An unknown model raises instead of guessing a route. Misconfiguration is therefore loud and early. build_agent validates every role against the catalog before it clones anything, naming all the bad ones at once. Discovery is per key -- /model/info returns only what the presented token may use -- so the same check catches a role pointed at a real model this key cannot reach. An unreachable proxy is reported separately, since an outage and a typo call for different responses from the operator. Two things now get recorded rather than inferred: the resolved model is pinned alongside the role mapping at task start, for the reason A11 already pins roles, and DelegationStarted carries backing_model so A16 eval history is not averaged across whatever an alias happened to point at. A model LiteLLM declares tool-less warns at startup rather than refusing; A15 expects weak local models to be accommodated, and they are observed driving tool loops anyway. Verified live: all four aliases the production key can see route correctly through discovery, and each chosen route works. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-23 01:47:58 +00:00
dongarandClaude Opus 5	8cfa116aa1	Preserve delegation work through any upstream outage, not just budget ... An exhausted budget was a special case of something more general: the work is not at fault. A network blip, a timeout, a 5xx from the proxy, or a rate limit that did not clear all leave a delegation branch that may be perfectly good, and discarding it would throw away work for a reason that has nothing to do with the work. UpstreamUnavailable now names that family, with BudgetExhausted a member of it — distinguishable where the difference matters, since a budget will not clear within the hour while a network blip will. A delegation stopped by any of them is "interrupted": branch kept, intent left pending, and A3's resume rule picks it up. The benefit of the doubt stops at the server's own door. A 4xx that is not a rate limit means the request itself was wrong, and preserving branches on the strength of one would hide a real bug — so those still fail and clean up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-22 23:57:05 +00:00
dongarandClaude Opus 5	8cfb45aed3	Enforce the token cap, and treat a spent budget as an interruption ... Exceeding a LiteLLM spend limit takes external models away for the rest of the day, which makes three existing gaps consequential rather than untidy. - A13 calls the delegation caps hard on "turns, tokens, and wall clock", and the module docstring repeated it, but the token cap was decoration: nothing counted, and result.tokens was always zero. It is now metered from the usage the provider reports, so a confused delegate cannot spend a day's allowance. A5 keeps the *money* cap on the LiteLLM key, which is a different thing; this is a scope cap - The normalized route sent no ceiling at all, so truncation detection there had nothing to detect against. Both routes now send one - A budget refusal is recognised distinctly. It is not a rate limit: retrying cannot help, so the planner no longer burns a second call to fail identically And a delegation cut off by an exhausted budget is now "interrupted" rather than "failed", which preserves its branch and leaves its intent pending. The work on that branch may be perfectly good; discarding it because the budget ran out would throw away work for a reason that has nothing to do with the work. A3's resume rule already knows how to pick such a delegation back up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-22 23:54:57 +00:00
dongarandClaude Opus 5	249bcb10fc	Detect truncated responses, and raise the output ceiling ... Measuring Opus found a bug that had been silent all along: nothing checked stop_reason or finish_reason, so a response cut off mid-answer was parsed as a complete one. The symptom was a submit_plan tool call whose required `content` field was simply absent — structurally valid, substantively empty, and indistinguishable from a model that chose to send nothing. The cause was our max_tokens of 4096: Opus writing an architecture plan routinely spends 3,000-4,000 output tokens, and was observed at 4,064 against that ceiling. Roughly one run in three crossed it. Worse than the empty plan is what the same failure does elsewhere: a truncated write_file writes half a file and reports success. - Both routes now raise TruncatedResponse when the model was cut off. It subclasses LLMError, so the planner's retry and the executor's and delegation's gating already handle it without learning a new type: a cut-off turn is a failed turn - The default ceiling is 16384 and configurable via NISSE_MAX_OUTPUT_TOKENS. Output is billed by what is produced, not by the ceiling, so a low limit buys nothing and costs silent truncation Verified: three consecutive Opus planning turns now return plans of 5,700-6,200 characters where one in three previously returned nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-22 23:45:28 +00:00
dongarandClaude Opus 5	ed2f5dfc7f	Choose the LLM route by model, and add a capability matrix ... The arrival of non-Anthropic models made a latent bug observable: the delegate's client was hardcoded to the Anthropic passthrough, so pointing NISSE_ROLE_DELEGATE at gpt-6-astra or a local model would have sent Anthropic-shaped /v1/messages bodies to something that does not speak them. A5 promises the frontier tier is swappable "as a configuration change, not a refactor", and it was not. select_client now picks the route from the model: the vendor-native adapter where one exists, LiteLLM's normalization otherwise — which is exactly what A13 prescribes for a vendor whose renderer adapter has not been written, pending validation against the A16 eval set. Also adds the capability matrix Part H asks for. It says the thresholds that depend on local capability — A12's failure count, A14's consult budget, A15's manifest size — must be tuned "on the real hardware", and that needs measurement rather than reputation. The harness probes each model for the things that actually break: whether it calls a tool at all, whether the plan it submits has substance (llama3.2 emitted "{"), whether it can consume its own tool result, and whether prompt-level JSON holds. Opt in with NISSE_CAPABILITY_MODELS; it spends real tokens across several models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-22 23:20:38 +00:00
dongarandClaude Opus 5	af9bdaac8d	Stop CI running two suites at once, and retry rate limits ... CI failed on the delegation branch with a 429 from the proxy: token limit 200000, 17399 remaining. Not a defect — the live delegation tests are token-hungry, and every push to a PR branch fired two identical suites concurrently, since the push and pull_request events both matched. - Push now only triggers on main. A PR branch is already covered by the pull_request event, so this halves both the wall time and the tokens spent per branch - A 429 is retried once before being surfaced. Losing a ten-turn delegation to a transient limit wastes far more than waiting out the window, and retrying forever would hide a real ceiling. The sleep is injectable, so the existing 429 test no longer waits 20 seconds for real Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-08 00:18:32 +00:00
dongarandClaude Opus 5	f4a94281ef	Add the Anthropic passthrough renderer and client (A10/A13) — 7a ... The second renderer adapter, and the concrete case for A13's decision to normalise in the renderer rather than on the wire. The two vendors disagree structurally, not cosmetically: the system prompt is a top-level field rather than a message, a tool call is a tool_use content block rather than an array, a tool result is a block inside a user message rather than a "tool" role, and tool schemas use input_schema rather than parameters. A middlebox flattening those would be lossy exactly where a long-running delegation lives. - render_anthropic.py builds vendor-native /v1/messages bodies from the canonical log, coalescing adjacent same-role blocks so a tool_result lands directly after its tool_use - Tool manifests are declared once in OpenAI shape and converted per vendor, so adding a tool does not mean editing two places - AnthropicPassthroughClient implements the same LLMClient protocol, so callers pick a route rather than a different kind of client, and maps provider tool ids to canonical ones at the boundary - Dangling tool calls get a placeholder result here too; this API rejects them more strictly than the OpenAI shape - Vendor-opaque reasoning blocks are dropped for now with the reason recorded: A10 wants them stored and never replayed across sessions, and the canonical schema has no event for that yet Live tests prove the body is accepted as sent, that a tool_result turn round-trips, and — as A5 asks — that prompt cache accounting survives the passthrough rather than being flattened by the proxy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 23:13:22 +00:00
dongarandClaude Opus 5	cf85648719	Add LLM-backed planning agent (A4/A11/A13) — step 5a ... - planner.py: LLMPlanningAgent offers exactly two tools, submit_plan and ask_human, so gating stays structural at the tool layer; no proceed tool exists. System prompt requires explicit, low-inference plans because a weaker model executes them (A13) - Model output is appended to the canonical log as it arrives, before being interpreted, so a crash mid-planning leaves a truthful transcript - A bad turn (prose instead of a tool call, malformed arguments, or a failed call) retries once, then raises PlanningError; the orchestrator gates to AWAIT_INPUT rather than inventing a plan - Orchestrator: appends every operator message, plan_submitted, and gate_opened to the transcript, so re-planning sees rejection feedback as conversation rather than out-of-band parameters; pins the A11 role mapping at task start - llm.py: non-JSON or non-object tool arguments now raise LLMError instead of escaping as JSONDecodeError — observed from llama3.2, which emits truncated argument strings - Live planning test is opt-in on NISSE_LIVE_PLANNER_MODEL: planning escalates unconditionally (A12) and the local model does not hold the tool-call contract, so it awaits a frontier route Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 14:47:12 +00:00
dongarandClaude Fable 5	0c2021fced	Add LLM client interface with LiteLLM adapter (A5/A11) ... PR 2 of the steps-3/4 plan. - llm.py: LLMClient protocol — complete(events, tools, model) -> canonical events — present from the first real LLM call. LiteLLMClient renders requests via the A10 renderer, targets the proxy's OpenAI-compatible route, and maps provider tool-call IDs to locally-minted canonical IDs at the boundary - RoleMap (A11): executor/planner/delegate/consultant resolved from env; pinned() returns the mapping to record at task start - Config strictly from env: NISSE_LLM_BASE_URL, NISSE_LLM_TOKEN; envfile.py is a minimal .env loader where real env wins; .env now gitignored with a committed .env.example documenting all variables (Matrix ones included for the upcoming transport PR) - Live smoke test against the proxy is integration-marked and skips without credentials — CI stays credential-free per operator decision; the test runs locally once .env exists - httpx added as nisse-dev's HTTP client Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-09-06 20:57:51 +00:00