dev/nisse
2
1 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Compare commits

base: dev/nisse:a17-knowledge-store
dev/nisse:main dev/nisse:a17-knowledge-store
dev/nisse:v0.1.0
..
compare: dev/nisse:570da5784e5a39d83500b1233af8b09d01e617cd
dev/nisse:main dev/nisse:a17-knowledge-store
dev/nisse:v0.1.0
49 Commits
a17-knowledge-store .. 570da5784e
Author SHA1 Message Date
dongarandClaude Opus 5.5 570da5784e Keep runs with different tuning apart in the report, and raise max_turns to 40
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m15s
Details 
The report grouped runs by model, serving note and task version, so
runs made at different tuning were silently averaged together. Tuning
is now part of the key, and a tuning column shows how a row's settings
differ from today's defaults; it appears only when some row needs it.

tuning.executor.max_turns goes from 24 to 40. On the harder benchmark
tasks Haiku, the calibration ceiling, hit 24 on both refactor-rename
runs and both long-milestone runs, one of which had finished the work.
The cap guards against a model trying forever; it isn't a budget.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-27 22:19:57 +00:00
dev-agent f41cbb00cf Merge pull request 'Record rate-limit waits in benchmark runs, and leave them out of time' (#50) from record-rate-limit-retries into main
CI / checks (push) Failing after 1m7s
Details
2026-09-27 21:23:56 +00:00
dongarandClaude Opus 5.5 4e45dfe06a Record rate-limit waits in benchmark runs, and leave them out of time
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m17s
Details 
The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.

Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-27 20:57:15 +00:00
dev-agent a716d5ec8d Merge pull request 'Benchmark seven new RTX 4090 models, and local-fast on the 4090' (#49) from bench-4090 into main
CI / checks (push) Successful in 1m8s
Details
2026-09-27 20:31:46 +00:00
dev-agent 1cc8df8dab Merge pull request 'Point the example config at Opus 5.5, with Haiku as consultant' (#48) from example-opus-5-5 into main
CI / checks (push) Successful in 1m7s
Details
2026-09-27 20:31:34 +00:00
dongarandClaude Opus 5.5 82f171c2f5 Benchmark seven new RTX 4090 models, and local-fast on the 4090
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m16s
Details 
Five runs per task for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-31b,
gemma-4-26b-a4b, gemma-4-12b, mimo-v2.6-9b and granite-4.2-8b, all
served by llama.cpp b11206 through llama-swap. Every one passed 20 of
20. local-fast (vLLM 0.30.0) passed 3 of 20.

Runs were made under a 350 W power cap, recorded in each note, after
the uncapped card throttled for heat. granite-4.2-8b also has one
uncapped set, kept apart by its note.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-27 18:43:55 +00:00
dongarandClaude Opus 5.5 3cb70c0656 Point the example config at Opus 5.5, with Haiku as consultant
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m13s
Details 
Planner and delegate use claude-opus-5-5 for the high-end judgement
those roles need. The consultant answers single questions with no tools,
so it uses claude-haiku-4-5. Sonnet has no role.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-25 20:22:49 +00:00
dongar 8a8d8dc27f Merge pull request 'Add an executor benchmark that keeps its results' (#47) from benchmark into main
CI / checks (push) Successful in 1m15s
Details 
Reviewed-on: dongar/nisse#47
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-25 02:23:57 +00:00
dongarandClaude Opus 5.5 0cccfe1e94 Add the local baseline: qwen3-8b-awq and qwen3-14b, 5 runs per task
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m14s
Details 
The 4070's two local models with default tuning, as the baseline the
4090's candidates will be compared with. Every one of the 22 failures
is an escalation from failure_limit: 2; there were no prose turns,
invented tools, context trims, or finished-but-wrong results.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-25 01:25:13 +00:00
dongarandClaude Opus 5.5 50b46821d7 Add an executor benchmark that keeps its results
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m11s
Details 
Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.

- Four canned milestones under nisse_dev/bench/tasks: write a function,
  fix a bug a failing test exposes, thread an option through several
  files, and find one failure in output far past max_output_chars. Each
  is an approved plan over a seed project, judged by a hidden check the
  model never sees; a reference solution proves each is solvable and
  that its check tells the difference
- Runs use the agent's own client wiring, now shared as
  make_llm_clients, with delegation and ask_expert off: an escalation is
  a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
  the model's backing name and declared facts, a note on how it was
  served, the task's content hash, the agent's commit and tuning, the
  outcome, turns, misbehaviour counts, context trims, tokens, time and
  LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
  95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
  a new model through the context probe, the capability matrix and the
  benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
  requests it had to trim
- The tuning guide gains "Benchmarking models"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-25 00:35:24 +00:00
dongar b2bbf70b7b Merge pull request 'Enforce the tool-call contract, asking servers only where it works' (#45) from enforce-tool-calls into main
CI / checks (push) Successful in 53s
Details 
Reviewed-on: dongar/nisse#45
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-25 00:05:46 +00:00
dongarandClaude Opus 5.5 b934c46714 Enforce the tool-call contract, asking servers only where it works
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 56s
Details 
The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.

- The catalog reads supports_tool_choice from LiteLLM's model_info;
  BudgetedClient sends tool_choice only to models declared true, as
  "required" on the normalized route and {"type": "any"} on the
  Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
  milestone. Finishing with nothing changed is refused once; a prose
  reply gets one nudge, then escalates; a call to a tool not offered
  counts toward failure_limit, like a failed command; calls after
  finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
  recognises a proxy fallback, which once credited Claude's answers to
  a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
  both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
  that models call only offered tools, and that a declared
  supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
  declared and backed up

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-24 22:54:49 +00:00
dongar 389b71a5d1 Merge pull request 'Fit every model request to its context window' (#44) from context-budget into main
CI / checks (push) Successful in 57s
Details 
Reviewed-on: dongar/nisse#44
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-24 13:41:15 +00:00
dongarandClaude Opus 5.5 494715efa9 Fit every model request to its context window
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s
Details 
Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.

- The model catalog reads max_input_tokens and max_output_tokens from
  LiteLLM's model_info, and startup refuses a role on a model with no
  declared window
- BudgetedClient wraps both routes. It fits the canonical events before
  rendering: older tool results cut to head and tail, then the oldest
  turns dropped with a note, keeping calls with their results; a request
  that still cannot fit fails with ContextOverflow and is never sent.
  The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
  estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
  the approved milestone plan, goes in its system prompt, and it sees
  only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
  failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
  server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
  and the probe

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-24 00:26:30 +00:00
dongar 31851b7396 Merge pull request 'Move configuration to YAML, and make the tuning knobs configurable' (#43) from yaml-config into main
CI / checks (push) Failing after 51s
Details 
Reviewed-on: dongar/nisse#43
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-23 23:49:44 +00:00
dongarandClaude Opus 5.5 e1034a2cb7 Show which config key refers to each example secret
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 49s
Details 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 23:45:02 +00:00
dongarandClaude Opus 5.5 58b439088f Pass the capability matrix's models as a pytest option
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 51s
Details 
NISSE_CAPABILITY_MODELS was neither a secret nor configuration: which
models to measure is a question for one run, so it is now an argument,
--capability-models=a,b, declared in tests/conftest.py where pytest
reads options at startup. With no models the matrix skips as before, so
CI is unchanged. The last non-secret NISSE_* variable other than
NISSE_CONFIG and NISSE_ENV is gone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 23:32:44 +00:00
dongarandClaude Opus 5.5 15d69799bf Make the config file the only home for settings
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 51s
Details 
With no production instance yet, there is no reason to carry two ways
of configuring the agent. Every environment variable that duplicated a
config key is gone; the environment now supplies only the secrets the
file references by ${…}, plus NISSE_CONFIG and NISSE_ENV to find the
files.

- AgentConfig loads from a file (load, from_file) or a parsed tree
  (from_tree, for tests); a missing file is an error naming the example
- The from_env constructors on LiteLLMClient, AnthropicPassthroughClient,
  GiteaClient, MatrixConfig and RoleMap are removed, with the env
  reading for max output tokens; clients take their settings from config
- The live integration tests share a committed agent config,
  tests/integration/nisse-test.yaml, aimed at test resources, and one
  conftest that loads it and builds clients. NISSE_LIVE_PLANNER_MODEL
  becomes its planner role; NISSE_CAPABILITY_MODELS stays as a per-run
  switch
- A missing secret now fails the live tests instead of skipping them
- CI passes only the three tokens
- .env.example is secrets only; spec A5 says config comes from the file

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 23:25:58 +00:00
dongarandClaude Opus 5.5 df449c7e34 Move configuration to a YAML file, and make the tuning knobs configurable
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 51s
Details 
The thresholds Part H says to tune on the real inference hardware were
hardcoded, so every adjustment would have been a PR. They now live in a
`tuning:` section of a YAML config file, which also replaces flat
environment variables as the home for structure (#32).

- The file (NISSE_CONFIG, default ./nisse-dev.yaml) carries structure
  and holds no secrets: credentials are ${VAR} references resolved from
  the environment, and a reference to an unset variable is an error
- The environment still wins: every existing NISSE_* variable overrides
  its key, and the file is optional, so .env and CI work unchanged
- Validation stays fail-fast and reports every problem at once; unknown
  keys are errors, so a misspelt knob fails at startup instead of
  silently doing nothing
- The config file joins the state store and .env in the check that
  refuses to start when they sit inside the workspace clone
- Tuning covers the executor's turn and failure limits, the consult
  budget, the delegation caps, the command timeout and the tool output
  limit, which Workspace now takes as a setting; the defaults are the
  values the agent already ran with
- llm.max_output_tokens reaches both LLM clients from the config

docs/Tuning-Guide.md explains the knobs for someone learning local
model tuning: how weights, quantization and the KV cache spend VRAM,
what each knob does and what goes wrong at either extreme, and an order
to tune in. nisse-dev.example.yaml documents every key and is checked
by a test so it cannot drift.

Closes #32

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 23:14:07 +00:00
dongar 4a9354ae3b Merge pull request 'Add the operator kill switch, stop (A4)' (#41) from stop-command into main
CI / checks (push) Successful in 50s
Details 
Reviewed-on: dongar/nisse#41
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-23 22:50:46 +00:00
dongarandClaude Opus 5.5 abb7fa0216 Add the operator kill switch, stop (A4)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 49s
Details 
`stop` could not have worked by adding the command alone. Planning
turns, milestones and review responses ran synchronously on the thread
that delivered the message, so the Matrix sync loop read nothing until
the work finished. The state store was also opened on the main thread
and used from the transport and poller threads, which SQLite refuses
with its default same-thread check; the in-memory test transport never
showed it.

A single worker thread now runs every handler and owns the store:

- The worker opens the store, so the same-thread check stays on and
  fails loudly if anything touches the store from elsewhere
- Matrix messages are handed to the worker; pollers hand over and wait,
  so their cursor never passes an unhandled comment
- A message that arrives during long work gets an immediate "busy; say
  'stop' to halt" reply and is recorded, not applied later in a state
  the operator was not answering
- Tests keep running inline through InlineWorker, which defers a job
  submitted from inside another as the real worker would queue it

`stop` sets a shared Cancel flag straight from the transport's thread.
The executor and delegation loops check it before each model call and
between tool calls; the planner checks it before each turn. Calls left
unrun get a result saying so, so the transcript still renders. A
stopped delegation is discarded like a failed one. The orchestrator
moves to AWAIT_INPUT and asks why the work was stopped. The reply is
recorded for the model, and a stopped planning turn resumes with it as
the answer. `stop` with nothing running says so and changes nothing; at
a gate it is no longer taken as rejection feedback.

Closes #40

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 22:47:29 +00:00
dongar d2fe3d5642 Merge pull request 'Gate ops writes on approved step sequences' (#39) from ops-gating into main
CI / checks (push) Successful in 1m8s
Details 
Reviewed-on: dongar/nisse#39
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-23 22:26:12 +00:00
dongarandClaude Opus 5.5 019fa24517 Wait a grace period before switching away from a waiting item
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 57s
Details 
Switching as soon as the active item posts a plan or question means the
operator sees the request and the switch notice seconds apart. A prompt
approval then waits until the other item waits or closes, which can be
a whole milestone. A configurable grace period (default 10 minutes)
lets a watching operator answer before the agent moves on, and still
switches when they are away.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 22:16:40 +00:00
dongarandClaude Opus 5.5 a91252cb77 Let the agent also switch on its own when the active item closes
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 57s
Details 
0.2 previously allowed an autonomous switch only while the active item
waits on the operator. Ready work should not sit behind an idle agent
either, so closing an item (completed or abandoned) is now the second
case. This also defines "ready": the agent can act on the item without
the operator, because it has not started or because intake arrived
while it was parked that answers what it was waiting for. A9's
work_item.switch allows a closed or null `from`, and B2 no longer says
only the operator selects the active project.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 22:09:39 +00:00
dongarandClaude Opus 5.5 5b163f0c2e Give calls outside any work item an agent-level scope
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 50s
Details 
Letting the model choose which ready item to switch to adds a model
call that belongs to no work item. Duty runs before a finding already
had the same gap. A10 now defines an agent-level scope for these calls,
kept apart from work item transcripts in both directions. The switch
call sees only code-rendered summaries of the ready items, runs as the
executor, and falls back to the oldest item if it fails. A9 gains a
work_item.switch event, and A18 accepts that an injected alert can at
worst reorder work.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 21:57:05 +00:00
dongarandClaude Opus 5.5 854bddc680 Let the model choose which ready work item to switch to
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 52s
Details 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 21:54:18 +00:00
dongarandClaude Opus 5.5 b8df5d2066 Gate ops writes on approved step sequences
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 52s
Details 
C2's per-tool allow/deny/confirm taxonomy would have to be written in
advance for a tool catalog that will grow large, and every mistake in it
is friction or a hole. An ops write plan is now a list of literal steps
(tool, target, arguments, optional precondition and expected outcome,
retry allowance), and one approval covers exactly that sequence:

- Code renders the approval message from the steps, flagging
  irreversible steps, protected targets, and steps that take down the
  agent's own host
- A matcher checks every call: read-only is free, an elevated call must
  exactly match the next approved step (arguments normalized against the
  tool's schema), and anything else halts to AWAIT_INPUT
- Approvals expire if the plan does not start in time
- Classification shrinks to a versioned policy file: instance privilege,
  protected targets, and each elevated tool's target argument, all
  failing safe; MCP annotations may only tighten
- Bounded matching and standing approvals are deferred, with reasons

Supporting changes:

- A3: an approved sequence resumes after the agent's own restart only on
  a verified expected outcome for the step in flight
- A4: an operator kill switch, `stop`, for every agent
- 0.2: the agent may switch items on its own while the active one waits
  on the operator; parking unmounts elevated instances; a Matrix reply
  to a parked item binds to that item
- A17: step preconditions replace the staleness bounds, the log-checked
  verification precondition and the boot-time re-fetch
- A18 (new): prompt injection, designed around limiting what a
  compromised model can do rather than detecting injection
- Part H records the decision and parks the deferred items

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 21:50:15 +00:00
dongar 182216296d Merge pull request 'Define the work model: duties, work items, scopes' (#38) from work-model into main
CI / checks (push) Successful in 1m1s
Details 
Reviewed-on: dongar/nisse#38
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-23 20:31:29 +00:00
dongarandClaude Opus 5.5 2c9a082e5b Define the work model: duties, work items, scopes
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 53s
Details 
"Project" was carrying several things at once: the repo, the effort
through arch plan and milestones, and the transcript's scope. The
Sysadmin Agent needs the same distinctions in a different shape, with
standing duties that produce many efforts over one scope. New section
0.2 names them:

- Duty: a standing responsibility that produces work items, and owns
  deduplication of its findings
- Work item: the unit of state, identified by a locally minted id
  (A9's task_id), optionally linked to an external record
- Scope: what a work item operates on, shared across work items
- Park-and-switch: one active work item, switching only at safe
  points, nothing lost while parked; the schema does not assume one

0.1 records this as a bounded exception to extract-after-it-works:
identity is designed ahead, behaviour is not. A2, A4, A9, A10, B1/B2
and C4 now speak in these terms; the transcript is scoped per work
item, and a milestone no longer mints a task_id. Part H parks
concurrency with its costs, and leaves open when a work item must
have an external record.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 20:26:24 +00:00
dongar 889615e3e2 Merge pull request 'Record the MCP server deployment decisions in the spec' (#36) from mcp-deployment into main
CI / checks (push) Successful in 54s
Details 
Reviewed-on: dongar/nisse#36
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-23 02:03:59 +00:00
dongarandClaude Opus 5.5 3dbfbc8cf0 Name the site domains the identifiers are stems of
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 54s
Details 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 02:00:43 +00:00
dongarandClaude Opus 5.5 cdc3881deb Fold the MCP deployment amendment into the spec
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m4s
Details 
Part F is restructured around rules that now apply to every agent:

- F1: every agent gets its own VM, for inter-agent isolation and
  per-agent credential attribution
- F2: a declared, per-instance default-deny egress allowlist that
  covers MCP servers' reach, with the tailnet's 100.64.0.0/10 called
  out
- F3: MCP servers per agent VM and per (service, site, privilege),
  plus the rejected routed-meta-tools alternative
- F4: the existing image pipeline, with third-party MCP images
  digest-pinned and excluded from Watchtower

Consequences land where they belong: site identifiers in Part 0, a
machine account per MCP server instance in A7, B3 and C5 recast as
instances of F1/F2, plan approval required for every write in C2, MCP
as the primary infrastructure path in C3, the network-gear browser
item narrowed to fallback access, and stateful-server relocation
parked in Part H.

The standalone amendment is removed now that its content lives here,
as with the Tiered Inference amendment; it remains in the previous
commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 01:56:30 +00:00
dongarandClaude Opus 5.5 31bef59151 Add the MCP deployment amendment 
Records a design session on MCP server deployment as a standalone
amendment, ahead of folding it into the spec:

- Every agent gets its own VM, justified by inter-agent isolation and
  per-agent credential attribution; Docker-in-Docker stays as an extra
  Dev Agent reason rather than the rule's foundation
- Egress is a declared, per-agent-instance default-deny allowlist,
  covering what MCP servers reach on the agent's behalf, with the
  tailnet's 100.64.0.0/10 called out as outside RFC1918
- MCP servers run per agent VM, one instance per (service, site,
  privilege), with site in the server identity rather than a tool
  argument
- Third-party MCP images are digest-pinned and excluded from Watchtower
- Routed meta-tools rejected; stateful-server relocation parked

Two consequences settled with the operator while drafting: each MCP
server instance gets its own Vaultwarden machine account (A7), and any
write, reversible or not, requires an approved plan (C2).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 01:55:31 +00:00
dongar b0d7adb70d Merge pull request 'Discover a model's vendor from LiteLLM instead of its name' (#35) from alias-discovery into main
CI / checks (push) Successful in 55s
Details 
Reviewed-on: dongar/nisse#35
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-23 01:49:36 +00:00
dongarandClaude Opus 5 e38adf40ca Discover a model's vendor from LiteLLM instead of its name
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 47s
Details 
Route selection sniffed model names: anything starting "claude-" or
"anthropic/" took the vendor-native passthrough, everything else took the
normalized route. That works only while an alias names its vendor. A
semantic role alias like "remote-architecture" is meant to point at
Anthropic today and somewhere else tomorrow, and the name says nothing
either way -- so it would have taken the normalized route regardless of
what was behind it, silently losing the fidelity A13 wants for delegations.

LiteLLM already knows the answer. /model/info reports each alias's backing
model, provider, and declared tool support, so the agent asks at startup and
routes on the provider it discovers. That keeps vendor knowledge where A5
already put it -- in the proxy -- rather than duplicating it as name patterns
here.

There is deliberately no name-based fallback: PASSTHROUGH_PREFIXES and
has_native_adapter are gone, and select_client now requires a catalog. A
second heuristic could only ever disagree with the proxy. An unknown model
raises instead of guessing a route.

Misconfiguration is therefore loud and early. build_agent validates every
role against the catalog before it clones anything, naming all the bad ones
at once. Discovery is per key -- /model/info returns only what the presented
token may use -- so the same check catches a role pointed at a real model
this key cannot reach. An unreachable proxy is reported separately, since an
outage and a typo call for different responses from the operator.

Two things now get recorded rather than inferred: the resolved model is
pinned alongside the role mapping at task start, for the reason A11 already
pins roles, and DelegationStarted carries backing_model so A16 eval history
is not averaged across whatever an alias happened to point at.

A model LiteLLM declares tool-less warns at startup rather than refusing;
A15 expects weak local models to be accommodated, and they are observed
driving tool loops anyway.

Verified live: all four aliases the production key can see route correctly
through discovery, and each chosen route works.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-23 01:47:58 +00:00
dongar 231b406393 Merge pull request 'Choose the LLM route by model, and add a capability matrix' (#34) from model-routing into main
CI / checks (push) Successful in 58s
Details 
Reviewed-on: dongar/nisse#34
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-23 01:47:34 +00:00
dongarandClaude Opus 5 8cfa116aa1 Preserve delegation work through any upstream outage, not just budget
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 52s
Details 
An exhausted budget was a special case of something more general: the
work is not at fault. A network blip, a timeout, a 5xx from the proxy,
or a rate limit that did not clear all leave a delegation branch that
may be perfectly good, and discarding it would throw away work for a
reason that has nothing to do with the work.

UpstreamUnavailable now names that family, with BudgetExhausted a
member of it — distinguishable where the difference matters, since a
budget will not clear within the hour while a network blip will. A
delegation stopped by any of them is "interrupted": branch kept,
intent left pending, and A3's resume rule picks it up.

The benefit of the doubt stops at the server's own door. A 4xx that is
not a rate limit means the request itself was wrong, and preserving
branches on the strength of one would hide a real bug — so those still
fail and clean up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-22 23:57:05 +00:00
dongarandClaude Opus 5 8cfb45aed3 Enforce the token cap, and treat a spent budget as an interruption
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 51s
Details 
Exceeding a LiteLLM spend limit takes external models away for the
rest of the day, which makes three existing gaps consequential rather
than untidy.

- A13 calls the delegation caps hard on "turns, tokens, and wall
  clock", and the module docstring repeated it, but the token cap was
  decoration: nothing counted, and result.tokens was always zero. It
  is now metered from the usage the provider reports, so a confused
  delegate cannot spend a day's allowance. A5 keeps the *money* cap on
  the LiteLLM key, which is a different thing; this is a scope cap
- The normalized route sent no ceiling at all, so truncation detection
  there had nothing to detect against. Both routes now send one
- A budget refusal is recognised distinctly. It is not a rate limit:
  retrying cannot help, so the planner no longer burns a second call
  to fail identically

And a delegation cut off by an exhausted budget is now "interrupted"
rather than "failed", which preserves its branch and leaves its intent
pending. The work on that branch may be perfectly good; discarding it
because the budget ran out would throw away work for a reason that has
nothing to do with the work. A3's resume rule already knows how to
pick such a delegation back up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-22 23:54:57 +00:00
dongarandClaude Opus 5 249bcb10fc Detect truncated responses, and raise the output ceiling
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 47s
Details 
Measuring Opus found a bug that had been silent all along: nothing
checked stop_reason or finish_reason, so a response cut off mid-answer
was parsed as a complete one.

The symptom was a submit_plan tool call whose required `content` field
was simply absent — structurally valid, substantively empty, and
indistinguishable from a model that chose to send nothing. The cause
was our max_tokens of 4096: Opus writing an architecture plan
routinely spends 3,000-4,000 output tokens, and was observed at 4,064
against that ceiling. Roughly one run in three crossed it.

Worse than the empty plan is what the same failure does elsewhere: a
truncated write_file writes half a file and reports success.

- Both routes now raise TruncatedResponse when the model was cut off.
  It subclasses LLMError, so the planner's retry and the executor's
  and delegation's gating already handle it without learning a new
  type: a cut-off turn is a failed turn
- The default ceiling is 16384 and configurable via
  NISSE_MAX_OUTPUT_TOKENS. Output is billed by what is produced, not
  by the ceiling, so a low limit buys nothing and costs silent
  truncation

Verified: three consecutive Opus planning turns now return plans of
5,700-6,200 characters where one in three previously returned nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-22 23:45:28 +00:00
dongarandClaude Opus 5 ed2f5dfc7f Choose the LLM route by model, and add a capability matrix
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 58s
Details 
The arrival of non-Anthropic models made a latent bug observable: the
delegate's client was hardcoded to the Anthropic passthrough, so
pointing NISSE_ROLE_DELEGATE at gpt-6-astra or a local model would
have sent Anthropic-shaped /v1/messages bodies to something that does
not speak them. A5 promises the frontier tier is swappable "as a
configuration change, not a refactor", and it was not.

select_client now picks the route from the model: the vendor-native
adapter where one exists, LiteLLM's normalization otherwise — which is
exactly what A13 prescribes for a vendor whose renderer adapter has
not been written, pending validation against the A16 eval set.

Also adds the capability matrix Part H asks for. It says the
thresholds that depend on local capability — A12's failure count,
A14's consult budget, A15's manifest size — must be tuned "on the real
hardware", and that needs measurement rather than reputation. The
harness probes each model for the things that actually break: whether
it calls a tool at all, whether the plan it submits has substance
(llama3.2 emitted "{"), whether it can consume its own tool result,
and whether prompt-level JSON holds. Opt in with
NISSE_CAPABILITY_MODELS; it spends real tokens across several models.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-22 23:20:38 +00:00
dongar 1da94d2581 Merge pull request 'Add the composition root and a runnable entry point' (#31) from agent-entrypoint into main
CI / checks (push) Successful in 1m13s
Details 
Reviewed-on: dongar/nisse#31
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-08 22:33:45 +00:00
dongarandClaude Opus 5 dfa574aeec Refuse to start with the state store or .env inside the clone
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m14s
Details 
Answering "is the workspace where the agent runs, or what it works
on?" — it is what it works on, and conflating the two is dangerous
rather than merely untidy.

The agent commits its workspace with git add -A and pushes it, so
anything inside the clone reaches the project repository. A .env
placed there would publish the LiteLLM, Matrix and Gitea tokens as a
side effect of the agent working correctly: silently, once, and
probably unnoticed. Startup now refuses both that and a state store
inside the clone, and .env.example documents the intended layout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-08 22:31:39 +00:00
dongarandClaude Opus 5 2fb25b8195 Add the composition root and a runnable entry point
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m14s
Details 
Nothing had ever assembled the agent. main.py was still the hello-world
placeholder from the scaffold, and no code outside the test suite
constructed an Orchestrator — which is how two fully built, fully
tested components came to be reachable from nothing: the delegation
harness, and the Gitea comment poller. Unit tests that construct a
component directly cannot catch that; only assembly can.

- config.py validates every variable before anything is built and
  names all of them at once, rather than failing at first use twenty
  minutes into a milestone on the call that mattered. Whitespace-only
  counts as missing, since a token of spaces passes a truthiness check
  and fails at the first request instead
- agent.py builds the whole graph and owns startup and shutdown order:
  reconcile before accepting commands, and on stop, stop watching,
  then stop talking, then close the store last
- The Gitea account is read via whoami rather than configured: it names
  the identity every permission is evaluated against, and proves the
  token works before anything depends on it
- Poller lifecycle is now a state-machine concern: watch the plan issue
  while a gate stands, the PR while it is open, and stop on leaving.
  The orchestrator owns when to watch; an injected factory owns how to
  build one
- The executor reports each turn, because it runs synchronously inside
  the transport callback: without it the agent is silent for a whole
  milestone and a working run looks exactly like a hung one
- main.py loads .env, builds, and shuts down cleanly on SIGINT/SIGTERM

The entry-point tests changed shape: nisse-dev no longer greets, it
refuses to start unconfigured and says everything that is missing. Both
tests now run with a deliberately empty environment and a .env path
that does not exist, because the working directory's .env holds live
credentials and a test must never be one cwd away from starting a real
agent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-08 22:26:26 +00:00
dongar 6331fcafd9 Merge pull request 'Wire delegation to A12's trigger, and implement A3's resume rule' (#25) from delegation-resume into main
CI / checks (push) Successful in 1m18s
Details 
Reviewed-on: dongar/nisse#25
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-08 21:46:26 +00:00
dongarandClaude Opus 5 ce436262f4 Wire delegation to A12's trigger, and implement A3's resume rule
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m8s
Details 
Two gaps found by auditing spec against code, the second created by
7b landing the harness without its boot-time half.

**The harness was not connected to anything.** Only a live test
constructed a Delegation; A12's failure-count trigger still gated to
the human, which A13 describes as the fallback rather than the path.
Consecutive command failures now hand the sub-task to the delegate,
with the failures in the brief so it need not rediscover what the
local model learned the hard way. A usable handback lets the local
loop carry on; anything else escalates, because a cap breach leaves a
half-edited tree.

**A3 step 5 was unimplemented.** A crash mid-delegation failed safe —
boot gated on "restarted mid-EXECUTING" — but always gated, even on
the clean cut the spec says should resume, and left the branch orphaned
because _abandon only runs while the process lives.

- A delegation now records an intent before the branch exists, and
  settles it whichever way the delegation ends; an aborted delegation
  is a finished one, and leaving it pending would make the next boot
  believe work was still in flight
- delegation_verdict implements the clean-cut test as an observable
  check, never model self-assessment: settled if a completion follows
  the start, absent if the branch was never created, gate on a
  tool_call with no recorded result, resume otherwise
- recover() sweeps at boot, announcing a resume with a notice rather
  than a question, since a clean cut needs no decision from the
  operator, and leaving an unclean one pending so the generic sweep
  gates on it
- resume() continues the same delegation rather than starting a
  second: no new branch, no second delegation_started, because A10
  defines a session as the delegation, not the process lifetime

Honest limit, stated in the code: "the branch sits at the expected
commit" is approximated by the branch existing and carrying the work,
since per-turn shas are not recorded in the log, so a branch rewritten
underneath us would go undetected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-08 20:03:46 +00:00
dongar 608a3a5bfb Merge pull request 'Add ask_expert and eval capture (A14/A16) — step 7c' (#24) from ask-expert into main
CI / checks (push) Successful in 1m9s
Details 
Reviewed-on: dongar/nisse#24
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-08 19:53:30 +00:00
dongarandClaude Opus 5 2b7651fb37 Add ask_expert and eval capture (A14/A16) — step 7c
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m7s
Details 
Completes step 7.

- expert.py: one stateless call for a question needing no repo access.
  Structured output is contracted at the prompt level, since vendors
  differ on JSON enforcement, so the reply is parsed and validated
  locally with one retry — tolerating a fenced code block, because
  models wrap JSON habitually, but not missing fields
- The per-task budget is counted from the canonical log rather than
  memory, so it survives a restart and cannot hand a circling executor
  a fresh allowance. Exhaustion raises rather than answering: a caller
  out of consults should get on with the work or escalate
- Consults are recorded as delegation_started/completed with role
  consultant. A16 asks for a "consult/delegation eval set", and
  reusing the composite events gets consult evaluation for free rather
  than inventing a parallel record
- evals.py pairs starts with completions into eval records, with
  verdicts kept outside the canonical log: the log records what
  happened, a verdict is a later judgement about it, and judgements
  get revised. Schema v3 adds the outcomes table, so a v1 store now
  traverses two migrations — a better exercise of the hook than one
- ask_expert is offered to the executor as a tool only when a desk is
  configured, in addition to the observable escalation triggers and
  never instead of them (A12). Consulting is not mutating, so a
  completion claim backed only by consults is still challenged

Found live: passing an empty message list and putting the question in
the system prompt fails outright — the Messages API requires at least
one non-system message. The question now travels as a synthesised user
message that is never appended to the log, which keeps the consult
stateless while remaining a valid request.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-08 19:48:13 +00:00
dongar dd22eda528 Merge pull request 'Map A17 onto Outline's actual primitives, and record the MCP decision' (#23) from a17-outline-mapping into main
CI / checks (push) Successful in 1m2s
Details 
Reviewed-on: dongar/nisse#23
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-08 19:38:29 +00:00
dongar f2ba932744 Merge pull request 'Add A17, the knowledge store contract (Outline)' (#22) from a17-knowledge-store into main
CI / checks (push) Successful in 59s
Details 
Reviewed-on: dongar/nisse#22
Reviewed-by: Don Garrett <dgarrett@acm.org>
 2026-09-08 19:25:00 +00:00
104 changed files with 9567 additions and 577 deletions
Expand all files Collapse all files

No files matched your search

.env.example
+15 -34
View File
@@ -1,36 +1,17 @@
# Copy to .env (chmod 600). Real environment variables win over this file.
# .env is gitignored; never commit real values.
# Secrets only. Copy to .env (chmod 600); it is gitignored, so never commit
# real values. Real environment variables win over this file.
#
# Everything else lives in the YAML config (see nisse-dev.example.yaml),
# which refers to these by name, e.g. `token: ${NISSE_LLM_TOKEN}`. The live
# integration tests use the same names, from tests/integration/nisse-test.yaml.
# --- LLM access (A5): the LiteLLM proxy is the single LLM destination ---
NISSE_LLM_BASE_URL=http://litellm.example:4000
NISSE_LLM_TOKEN=sk-your-litellm-virtual-key
# llm.token: the LiteLLM virtual key (A5)
NISSE_LLM_TOKEN=replace-me
# matrix.token: the Matrix access token for the agent's account (A1)
NISSE_MATRIX_TOKEN=replace-me
# gitea.token: the Gitea service account's scoped token (B2)
NISSE_GITEA_TOKEN=replace-me
# --- Role mapping (A11): roles resolve to concrete model strings ---
NISSE_ROLE_EXECUTOR=anthropic/claude-haiku-4-5-20251001
NISSE_ROLE_PLANNER=anthropic/claude-opus-5
NISSE_ROLE_DELEGATE=anthropic/claude-opus-5
NISSE_ROLE_CONSULTANT=anthropic/claude-opus-5
# Opt-in for the live planning test: a model the virtual key can reach that
# holds the tool-call contract. Planning escalates unconditionally (A12), and
# small local models do not hold the contract reliably — llama3.2 returns
# truncated tool arguments, Haiku-class and up are solid.
NISSE_LIVE_PLANNER_MODEL=claude-haiku-4-5
# --- Matrix transport (A1), used by nisse-dev's MatrixTransport ---
NISSE_MATRIX_HOMESERVER=https://matrix.example
NISSE_MATRIX_USER_ID=@nisse-dev:example
NISSE_MATRIX_TOKEN=syt_your_matrix_access_token
NISSE_MATRIX_ROOM_ID=!roomid:example
# Dedicated room for the live integration test; never the production room.
NISSE_MATRIX_TEST_ROOM_ID=!testroomid:example
NISSE_MATRIX_OPERATORS=@you:example
# --- Gitea (B2): service account for branches, PRs, and comment intake ---
NISSE_GITEA_URL=https://gitea.example
NISSE_GITEA_TOKEN=your-scoped-service-account-token
# Sandbox repo for live tests; never the real project repo.
NISSE_GITEA_TEST_REPO=owner/sandbox-repo
# Gitea logins allowed to drive the agent via PR comments (A4 per-channel
# allowlist; a Matrix MXID and a Gitea login are different names).
NISSE_GITEA_OPERATORS=your-gitea-login
# Where the config file is, if not ./nisse-dev.yaml. Where this file is, if
# not ./.env, is NISSE_ENV, which has to be set in the real environment.
# NISSE_CONFIG=~/nisse-demo-workspace/nisse-dev.yaml
.gitea/workflows/ci.yml
+3 -11
View File
@@ -34,19 +34,11 @@ jobs:
run: uv run pytest -m "not integration"
- name: Integration tests
# Live-path credentials come from repo Actions secrets; the tests
# skip-guard on these, so a missing secret or unreachable service
# degrades to "skipped", never a failure on unrelated work.
# Everything but the secrets is in tests/integration/nisse-test.yaml,
# which refers to these by name. A missing secret fails the live
# tests that need it; it is never quietly turned into a skip.
env:
NISSE_LLM_BASE_URL: ${{ secrets.NISSE_LLM_BASE_URL }}
NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }}
NISSE_ROLE_EXECUTOR: ${{ secrets.NISSE_ROLE_EXECUTOR }}
NISSE_LIVE_PLANNER_MODEL: ${{ secrets.NISSE_LIVE_PLANNER_MODEL }}
NISSE_MATRIX_HOMESERVER: ${{ secrets.NISSE_MATRIX_HOMESERVER }}
NISSE_MATRIX_USER_ID: ${{ secrets.NISSE_MATRIX_USER_ID }}
NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }}
NISSE_MATRIX_TEST_ROOM_ID: ${{ secrets.NISSE_MATRIX_TEST_ROOM_ID }}
NISSE_GITEA_URL: ${{ secrets.NISSE_GITEA_URL }}
NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }}
NISSE_GITEA_TEST_REPO: ${{ secrets.NISSE_GITEA_TEST_REPO }}
run: uv run pytest -m integration -rs
.gitignore
+1
View File
@@ -6,3 +6,4 @@ __pycache__/
.pytest_cache/
.ruff_cache/
dist/
nisse-dev.yaml
bench/results/.gitkeep
View File
Whitespace-only changes.
bench/results/2026-09-25-local-coder.jsonl
+20
View File
@@ -0,0 +1,20 @@
{"backing": "openai/qwen3-14b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Fixed total_value to multiply unit price by quantity. All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5328, "tokens_out": 1492, "trimmed_requests": 0, "ts": "2026-09-25T01:03:24+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 46.4}
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\nImportError while importing test module '/tmp/tmpr1q1pe0r/work/.bench_check/test_hidden.py'.\nHint: make sure your test modules/packages have valid Python names.\nTraceback:\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n.bench_check/test_hidden.py:4: in <module>\n from inventory import Item, low_stock, total_value\nE ImportError: cannot import name 'Item' from 'inventory' (/tmp/tmpr1q1pe0r/work/inventory.py)\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.10s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n______________________ ERROR collecting test_inventory.py ______________________\nImportError while importing test module '/tmp/tmpr1q1pe0r/work/test_inventory.py'.\nHint: make sure your test modules/packages have valid Python names.\nTraceback:\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], packa", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 2560, "tokens_out": 1136, "trimmed_requests": 0, "ts": "2026-09-25T01:03:49+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 24.3}
{"backing": "openai/qwen3-14b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Fixed the total_value function to multiply unit price by quantity, passing all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5318, "tokens_out": 1623, "trimmed_requests": 0, "ts": "2026-09-25T01:04:23+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 34.5}
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n.bench_check/test_hidden.py:4: in <module>\n from inventory import Item, low_stock, total_value\ninventory.py:1: in <module>\n def total_value(items: list[Item]) -> float:\n ^^^^\nE NameError: name 'Item' is not defined\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py - NameError: name 'Item' is not defined\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.11s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n______________________ ERROR collecting test_inventory.py ______________________\ntest_inventory.py:1: in <module>\n from inventory import Item, low_stock, total_value\ninventory.py:1: in <module>\n def total_value(items: list[Item]) -> float:\n ^^^^\nE NameError: name 'Item' is not defined\n=========================== short ", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 2565, "tokens_out": 958, "trimmed_requests": 0, "ts": "2026-09-25T01:04:45+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 20.6}
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n.bench_check/test_hidden.py:4: in <module>\n from inventory import Item, low_stock, total_value\ninventory.py:1: in <module>\n def total_value(items: list[Item]) -> float:\n ^^^^\nE NameError: name 'Item' is not defined\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py - NameError: name 'Item' is not defined\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.09s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n______________________ ERROR collecting test_inventory.py ______________________\ntest_inventory.py:1: in <module>\n from inventory import Item, low_stock, total_value\ninventory.py:1: in <module>\n def total_value(items: list[Item]) -> float:\n ^^^^\nE NameError: name 'Item' is not defined\n=========================== short ", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 2565, "tokens_out": 1051, "trimmed_requests": 0, "ts": "2026-09-25T01:05:07+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 22.6}
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmpq8iz1j8z/work/rates.py\", line 56\nE [... 1445 characters omitted ...]\nE ^^^^^^^^\nE SyntaxError: invalid syntax. Perhaps you forgot a comma?\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 13616, "tokens_out": 3186, "trimmed_requests": 0, "ts": "2026-09-25T01:06:21+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 72.8}
{"backing": "openai/qwen3-14b", "check_output": ": 'R271'\nFAILED .bench_check/test_hidden.py::test_rate[272] - KeyError: 'R272'\nFAILED .bench_check/test_hidden.py::test_rate[273] - KeyError: 'R273'\nFAILED .bench_check/test_hidden.py::test_rate[274] - KeyError: 'R274'\nFAILED .bench_check/test_hidden.py::test_rate[275] - KeyError: 'R275'\nFAILED .bench_check/test_hidden.py::test_rate[276] - KeyError: 'R276'\nFAILED .bench_check/test_hidden.py::test_rate[277] - KeyError: 'R277'\nFAILED .bench_check/test_hidden.py::test_rate[278] - KeyError: 'R278'\nFAILED .bench_check/test_hidden.py::test_rate[279] - KeyError: 'R279'\nFAILED .bench_check/test_hidden.py::test_rate[280] - KeyError: 'R280'\nFAILED .bench_check/test_hidden.py::test_rate[281] - KeyError: 'R281'\nFAILED .bench_check/test_hidden.py::test_rate[282] - KeyError: 'R282'\nFAILED .bench_check/test_hidden.py::test_rate[283] - KeyError: 'R283'\nFAILED .bench_check/test_hidden.py::test_rate[284] - KeyError: 'R284'\nFAILED .bench_check/test_hidden.py::test_rate[285] - KeyError: 'R285'\nFAILED .bench_check/test_hidden.py::test_rate[286] - KeyError: 'R286'\nFAILED .bench_check/test_hidden.py::test_rate[287] - KeyError: 'R287'\nFAILED .bench_check/test_hidden.py::test_rate[288] - KeyError: 'R288'\nFAILED .bench_check/test_hidden.py::test_rate[289] - KeyError: 'R289'\nFAILED .bench_check/test_hidden.py::test_rate[290] - KeyError: 'R290'\nFAILED .bench_check/test_hidden.py::test_rate[291] - KeyError: 'R291'\nFAILED .bench_check/test_hidden.py::test_rate[292] - KeyError: 'R292'\nFAILED .bench_check/test_hidden.py::test_rate[293] - KeyError: 'R293'\nFAILED .bench_check/test_hidden.py::test_rate[294] - KeyError: 'R294'\nFAILED .bench_check/test_hidden.py::test_rate[295] - KeyError: 'R295'\nFAILED .bench_check/test_hidden.py::test_rate[296] - KeyError: 'R296'\nFAILED .bench_check/test_hidden.py::test_rate[297] - KeyError: 'R297'\nFAILED .bench_check/test_hidden.py::test_rate[298] - KeyError: 'R298'\nFAILED .bench_check/test_hidden.py::test_rate[299] - KeyError: 'R299'\n247 failed, 54 passed in 0.49s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n.....................................................FFFFFFFFFFFFFFFFFFF [ 24%]\nFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [ 48%]\nFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [ 72%]\nFFFFFFFFFFFFFFFFFFFFF.FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [ 96%]\nFFFFFFFFFFFF [100%]\n=================================== FAILU", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 12726, "tokens_out": 2366, "trimmed_requests": 0, "ts": "2026-09-25T01:07:16+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 54.4}
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmphac71897/work/rates.py\", line 56\nE [... 1445 characters omitted ...]\nE ^^^^^^^^\nE SyntaxError: invalid syntax. Perhaps you forgot a comma?\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.15s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 13614, "tokens_out": 3000, "trimmed_requests": 0, "ts": "2026-09-25T01:08:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 69.2}
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmp0vs0w4kr/work/rates.py\", line 56\nE [... 1445 characters omitted ...]\nE ^^^^^^^^\nE SyntaxError: invalid syntax. Perhaps you forgot a comma?\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 13607, "tokens_out": 3223, "trimmed_requests": 0, "ts": "2026-09-25T01:09:40+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 73.8}
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmp_pxm0ldr/work/rates.py\", line 56\nE [... 1445 characters omitted ...]\nE ^^^^^^^^\nE SyntaxError: invalid syntax. Perhaps you forgot a comma?\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.15s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 13609, "tokens_out": 3180, "trimmed_requests": 0, "ts": "2026-09-25T01:10:53+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 73.0}
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n..FF [100%]\n=================================== FAILURES ===================================\n_________________________ test_cli_prints_the_greeting _________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x777eb60e3e90>\n\n def test_cli_prints_the_greeting(capsys) -> None:\n> assert main([\"Bob\"])\nE AssertionError: assert 0\nE + wh", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 13753, "tokens_out": 5755, "trimmed_requests": 0, "ts": "2026-09-25T01:12:55+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 121.2}
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added --shout option to greeter, updated core and cli code, added tests, and verified with pytest.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 21373, "tokens_out": 9952, "trimmed_requests": 0, "ts": "2026-09-25T01:16:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 13, "wall_s": 210.0}
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n..FF [100%]\n=================================== FAILURES ===================================\n_________________________ test_cli_prints_the_greeting _________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x7a2788fdbe30>\n\n def test_cli_prints_the_greeting(capsys) -> None:\n> assert main([\"Bob\"])\nE AssertionError: assert 0\nE + wh", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9432, "tokens_out": 6691, "trimmed_requests": 0, "ts": "2026-09-25T01:18:47+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 142.1}
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added --shout option to greeter, updated core and cli code, added tests, and verified with pytest.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9104, "tokens_out": 3626, "trimmed_requests": 0, "ts": "2026-09-25T01:20:03+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 75.8}
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n..FF [100%]\n=================================== FAILURES ===================================\n_________________________ test_cli_prints_the_greeting _________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x76eac82e3da0>\n\n def test_cli_prints_the_greeting(capsys) -> None:\n> assert main([\"Bob\"])\nE AssertionError: assert 0\nE + wh", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9382, "tokens_out": 7755, "trimmed_requests": 0, "ts": "2026-09-25T01:22:48+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 164.5}
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function and passed all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 1769, "tokens_out": 1300, "trimmed_requests": 0, "ts": "2026-09-25T01:23:16+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 27.7}
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function with tests; all tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 1782, "tokens_out": 1239, "trimmed_requests": 0, "ts": "2026-09-25T01:23:43+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 26.2}
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function and passed all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3627, "tokens_out": 1332, "trimmed_requests": 0, "ts": "2026-09-25T01:24:11+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 28.1}
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function and passed all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 734, "tokens_out": 653, "trimmed_requests": 0, "ts": "2026-09-25T01:24:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 1, "wall_s": 13.9}
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function and passed all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 733, "tokens_out": 523, "trimmed_requests": 0, "ts": "2026-09-25T01:24:37+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 1, "wall_s": 11.3}
bench/results/2026-09-25-local-fast.jsonl
+20
View File
@@ -0,0 +1,20 @@
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1578, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-25T01:00:41+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 63.2}
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1576, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-25T01:00:43+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 1.4}
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1576, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-25T01:00:44+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 1.4}
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1574, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-25T01:00:46+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 1.4}
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1578, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-25T01:00:48+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 1.4}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmpl4mdto5g/work/rates.py\", line 1\nE \"R237\": 0.12,\nE IndentationError: unexpected indent\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 11000, "tokens_out": 183, "trimmed_requests": 0, "ts": "2026-09-25T01:00:52+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 4.2}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmpsqvx_94y/work/rates.py\", line 1\nE \"R237\": 0.12,\nE IndentationError: unexpected indent\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 11000, "tokens_out": 183, "trimmed_requests": 0, "ts": "2026-09-25T01:00:57+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 4.2}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmpe3eakwxj/work/rates.py\", line 1\nE \"R237\": 0.12,\nE IndentationError: unexpected indent\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 11013, "tokens_out": 183, "trimmed_requests": 0, "ts": "2026-09-25T01:01:01+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 4.2}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmprlm2camm/work/rates.py\", line 1\nE \"R237\": 0.12,\nE IndentationError: unexpected indent\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 10987, "tokens_out": 183, "trimmed_requests": 0, "ts": "2026-09-25T01:01:06+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 4.2}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmpvzw7f5tn/work/rates.py\", line 1\nE \"R237\": 0.12,\nE IndentationError: unexpected indent\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 11000, "tokens_out": 183, "trimmed_requests": 0, "ts": "2026-09-25T01:01:11+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 4.2}
{"backing": "openai/qwen3-8b-awq", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n...F [100%]\n=================================== FAILURES ===================================\n_________________________ test_cli_prints_the_greeting _________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x713c88ce3c20>\n\n def test_cli_prints_the_greeting(capsys) -> None:\n> assert main([\"Bob\"])\nE AssertionError: assert 0\nE + wh", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 13039, "tokens_out": 566, "trimmed_requests": 0, "ts": "2026-09-25T01:01:19+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 8.4}
{"backing": "openai/qwen3-8b-awq", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n..FF [100%]\n=================================== FAILURES ===================================\n________________________________ test_cli_shout ________________________________\n\n def test_cli_shout() -> None:\n> from pytest import capsys\nE ImportError: cannot import name 'capsys' from 'pytest' (/root/nisse/.venv/lib/python3.12/site-packages/pytest/__init__.py)", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 11243, "tokens_out": 715, "trimmed_requests": 0, "ts": "2026-09-25T01:01:29+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 9.9}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:4: in <module>\n from greeter.cli import main\nE File \"/tmp/tmp3f09u7jk/work/greeter/cli.py\", line 11\nE print(greet(args.name, shout=args.--shout))\nE ^\nE SyntaxError: invalid syntax\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n\nstderr:\nTraceback (most recent call last):\n File \"<frozen runpy>\", line 198, in _run_module_as_main\n File \"<frozen runpy>\", line 88, in _run_code\n File \"/tmp/tmp3f09u7jk/work/greeter/__main__.py\", line 1, in <module>\n from greeter.cli import main\n File \"/tmp/tmp3f09u7jk/work/greeter/cli.py\", line 11\n print(greet(args.name, shout=args.--shout))\n ^\nSyntaxError: invalid syntax\n", "prose_turns": 0, "provider": "openai", "success": false, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9358, "tokens_out": 450, "trimmed_requests": 0, "ts": "2026-09-25T01:01:56+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 26.5}
{"backing": "openai/qwen3-8b-awq", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "completed", "outcome_note": "Added --shout option to greeter. Tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 10784, "tokens_out": 549, "trimmed_requests": 0, "ts": "2026-09-25T01:02:04+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 7.5}
{"backing": "openai/qwen3-8b-awq", "check_output": " stderr arguments may not be used '\n 'with capture_output.')\n kwargs['stdout'] = PIPE\n kwargs['stderr'] = PIPE\n \n with Popen(*popenargs, **kwargs) as process:\n try:\n stdout, stderr = process.communicate(input, timeout=timeout)\n except TimeoutExpired as exc:\n process.kill()\n if _mswindows:\n # Windows accumulates the output in a single blocking\n # read() call run on child threads, with the timeout\n # being done in a join() on those threads. communicate()\n # _after_ kill() is required to collect that and add it\n # to the exception.\n exc.stdout, exc.stderr = process.communicate()\n else:\n # POSIX _communicate already populated the output so\n # far into the TimeoutExpired exception.\n process.wait()\n raise\n except: # Including KeyboardInterrupt, communicate handled that.\n process.kill()\n # We don't call process.wait() as .__exit__ does that for us.\n raise\n retcode = process.poll()\n if check and retcode:\n> raise CalledProcessError(retcode, process.args,\n output=stdout, stderr=stderr)\nE subprocess.CalledProcessError: Command '['/root/nisse/.venv/bin/python', '-m', 'greeter', '--shout', 'Bob']' returned non-zero exit status 2.\n\n/usr/lib/python3.12/subprocess.py:571: CalledProcessError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_cli_default - SystemExit: 2\nFAILED .bench_check/test_hidden.py::test_cli_shout - SystemExit: 2\nFAILED .bench_check/test_hidden.py::test_module_entry_point - subprocess.Call...\n3 failed, 2 passed in 0.19s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n..F [100%]\n=================================== FAILURES ===================================\n________________________________ test_cli_shout ________________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x763f352e3980>\n\n def test_cli_shout(capsys) -> None:\n> assert main([\"--shout\", \"Bob\"]) == 0\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\ntest", "prose_turns": 0, "provider": "openai", "success": false, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 13124, "tokens_out": 589, "trimmed_requests": 0, "ts": "2026-09-25T01:02:13+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 8.4}
{"backing": "openai/qwen3-8b-awq", "check_output": " (\"Hello, World!\", \"hello-world\"),\n (\" Nisse Agent \", \"nisse-agent\"),\n (\"a--b\", \"a-b\"),\n (\"\", \"\"),\n (\"Python 3.12 Release\", \"python-3-12-release\"),\n (\"--leading and trailing--\", \"leading-and-trailing\"),\n (\"MiXeD CaSe\", \"mixed-case\"),\n (\"!!!\", \"\"),\n ],\n )\n def test_slugify(text: str, expected: str) -> None:\n> assert slugify(text) == expected\nE AssertionError: assert '' == 'leading-and-trailing'\nE \nE - leading-and-trailing\n\n.bench_check/test_hidden.py:19: AssertionError\n_____________________ test_slugify[MiXeD CaSe-mixed-case] ______________________\n\ntext = 'MiXeD CaSe', expected = 'mixed-case'\n\n @pytest.mark.parametrize(\n (\"text\", \"expected\"),\n [\n (\"Hello, World!\", \"hello-world\"),\n (\" Nisse Agent \", \"nisse-agent\"),\n (\"a--b\", \"a-b\"),\n (\"\", \"\"),\n (\"Python 3.12 Release\", \"python-3-12-release\"),\n (\"--leading and trailing--\", \"leading-and-trailing\"),\n (\"MiXeD CaSe\", \"mixed-case\"),\n (\"!!!\", \"\"),\n ],\n )\n def test_slugify(text: str, expected: str) -> None:\n> assert slugify(text) == expected\nE AssertionError: assert '' == 'mixed-case'\nE \nE - mixed-case\n\n.bench_check/test_hidden.py:19: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_slugify[Hello, World!-hello-world]\nFAILED .bench_check/test_hidden.py::test_slugify[ Nisse Agent -nisse-agent]\nFAILED .bench_check/test_hidden.py::test_slugify[a--b-a-b] - AssertionError: ...\nFAILED .bench_check/test_hidden.py::test_slugify[Python 3.12 Release-python-3-12-release]\nFAILED .bench_check/test_hidden.py::test_slugify[--leading and trailing---leading-and-trailing]\nFAILED .bench_check/test_hidden.py::test_slugify[MiXeD CaSe-mixed-case] - Ass...\n6 failed, 2 passed in 0.03s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF [100%]\n=================================== FAILURES ===================================\n_________________________________ test_slugify _________________________________\n\n def test_slugify():\n> assert slugify('Hello, World!') == 'hello-world'\nE AssertionError: assert '' == 'hello-world'\nE \nE - hello-world\n\ntest_slugify.py:6: AssertionEr", "prose_turns": 0, "provider": "openai", "success": false, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 4942, "tokens_out": 377, "trimmed_requests": 0, "ts": "2026-09-25T01:02:19+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 5.5}
{"backing": "openai/qwen3-8b-awq", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF [100%]\n=================================== FAILURES ===================================\n_________________________________ test_slugify _________________________________\n\n def test_slugify():\n> assert slugify('Hello, World!') == 'hello-world'\n ^^^^^^^\nE NameError: name 'slugify' is not defined\n\ntest_slugify.py:4: NameError\n===============", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3688, "tokens_out": 255, "trimmed_requests": 0, "ts": "2026-09-25T01:02:23+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 3.9}
{"backing": "openai/qwen3-8b-awq", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF [100%]\n=================================== FAILURES ===================================\n_________________________________ test_slugify _________________________________\n\n def test_slugify():\n> assert slugify('Hello, World!') == 'hello-world'\n ^^^^^^^\nE NameError: name 'slugify' is not defined\n\ntest_slugify.py:4: NameError\n===============", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3696, "tokens_out": 255, "trimmed_requests": 0, "ts": "2026-09-25T01:02:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 3.9}
{"backing": "openai/qwen3-8b-awq", "check_output": " (\"Hello, World!\", \"hello-world\"),\n (\" Nisse Agent \", \"nisse-agent\"),\n (\"a--b\", \"a-b\"),\n (\"\", \"\"),\n (\"Python 3.12 Release\", \"python-3-12-release\"),\n (\"--leading and trailing--\", \"leading-and-trailing\"),\n (\"MiXeD CaSe\", \"mixed-case\"),\n (\"!!!\", \"\"),\n ],\n )\n def test_slugify(text: str, expected: str) -> None:\n> assert slugify(text) == expected\nE AssertionError: assert '' == 'leading-and-trailing'\nE \nE - leading-and-trailing\n\n.bench_check/test_hidden.py:19: AssertionError\n_____________________ test_slugify[MiXeD CaSe-mixed-case] ______________________\n\ntext = 'MiXeD CaSe', expected = 'mixed-case'\n\n @pytest.mark.parametrize(\n (\"text\", \"expected\"),\n [\n (\"Hello, World!\", \"hello-world\"),\n (\" Nisse Agent \", \"nisse-agent\"),\n (\"a--b\", \"a-b\"),\n (\"\", \"\"),\n (\"Python 3.12 Release\", \"python-3-12-release\"),\n (\"--leading and trailing--\", \"leading-and-trailing\"),\n (\"MiXeD CaSe\", \"mixed-case\"),\n (\"!!!\", \"\"),\n ],\n )\n def test_slugify(text: str, expected: str) -> None:\n> assert slugify(text) == expected\nE AssertionError: assert '' == 'mixed-case'\nE \nE - mixed-case\n\n.bench_check/test_hidden.py:19: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_slugify[Hello, World!-hello-world]\nFAILED .bench_check/test_hidden.py::test_slugify[ Nisse Agent -nisse-agent]\nFAILED .bench_check/test_hidden.py::test_slugify[a--b-a-b] - AssertionError: ...\nFAILED .bench_check/test_hidden.py::test_slugify[Python 3.12 Release-python-3-12-release]\nFAILED .bench_check/test_hidden.py::test_slugify[--leading and trailing---leading-and-trailing]\nFAILED .bench_check/test_hidden.py::test_slugify[MiXeD CaSe-mixed-case] - Ass...\n6 failed, 2 passed in 0.03s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF [100%]\n=================================== FAILURES ===================================\n_________________________________ test_slugify _________________________________\n\n def test_slugify():\n> assert slugify('Hello, World!') == 'hello-world'\nE AssertionError: assert '' == 'hello-world'\nE \nE - hello-world\n\ntest_slugify.py:6: AssertionEr", "prose_turns": 0, "provider": "openai", "success": false, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 4936, "tokens_out": 377, "trimmed_requests": 0, "ts": "2026-09-25T01:02:33+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 5.5}
{"backing": "openai/qwen3-8b-awq", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.29, Qwen3-8B-AWQ, fp8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF [100%]\n=================================== FAILURES ===================================\n_________________________________ test_slugify _________________________________\n\n def test_slugify():\n> assert slugify('Hello, World!') == 'hello-world'\n ^^^^^^^\nE NameError: name 'slugify' is not defined\n\ntest_slugify.py:4: NameError\n===============", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3686, "tokens_out": 251, "trimmed_requests": 0, "ts": "2026-09-25T01:02:37+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 3.9}
bench/results/2026-09-27-gemma-4-12b.jsonl
+20
View File
@@ -0,0 +1,20 @@
{"backing": "openai/gemma-4-12b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to multiply `unit_price` by `quantity` for each item. All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 4909, "tokens_out": 853, "trimmed_requests": 0, "ts": "2026-09-27T17:32:39+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 11.1}
{"backing": "openai/gemma-4-12b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly multiply `unit_price` by `quantity` for each item. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5927, "tokens_out": 832, "trimmed_requests": 0, "ts": "2026-09-27T17:32:51+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 11.9}
{"backing": "openai/gemma-4-12b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the bug in `total_value` where it was only summing the unit price instead of multiplying by the quantity.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5927, "tokens_out": 623, "trimmed_requests": 0, "ts": "2026-09-27T17:33:01+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 9.8}
{"backing": "openai/gemma-4-12b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly multiply each item's unit price by its quantity.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5651, "tokens_out": 749, "trimmed_requests": 0, "ts": "2026-09-27T17:33:13+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 11.0}
{"backing": "openai/gemma-4-12b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly multiply `unit_price` by `quantity` for each item.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5927, "tokens_out": 704, "trimmed_requests": 0, "ts": "2026-09-27T17:33:24+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 10.6}
{"backing": "openai/gemma-4-12b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py from 0.21 to 0.12.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 18498, "tokens_out": 443, "trimmed_requests": 0, "ts": "2026-09-27T17:33:33+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 8.9}
{"backing": "openai/gemma-4-12b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect rate for region code R237 in `rates.py`. The rate was 0.21, but it should have been 0.12 as per the formula (NNN mod 25) / 100. All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 43881, "tokens_out": 8995, "trimmed_requests": 0, "ts": "2026-09-27T17:35:23+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 110.0}
{"backing": "openai/gemma-4-12b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect rate for region code R237 in rates.py. The expected rate was 0.12 (237 mod 25 / 100), but it was 0.21. All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 45997, "tokens_out": 9229, "trimmed_requests": 0, "ts": "2026-09-27T17:37:16+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 112.2}
{"backing": "openai/gemma-4-12b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect rate for region code R237 in rates.py, changing it from 0.21 to 0.12 as required by the test.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 43881, "tokens_out": 8879, "trimmed_requests": 0, "ts": "2026-09-27T17:39:04+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 107.8}
{"backing": "openai/gemma-4-12b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect rate for region R237 in rates.py. The rate was 0.21, but it should have been 0.12 (calculated as (237 mod 25) / 100 = 12 / 100 = 0.12). All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 46010, "tokens_out": 9357, "trimmed_requests": 0, "ts": "2026-09-27T17:40:58+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 113.5}
{"backing": "openai/gemma-4-12b", "check_output": "..... [100%]\n5 passed in 0.05s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a `--shout` option to the greeter. Updated `greeter/core.py` to support shouting, `greeter/cli.py` to include the flag, and added corresponding tests to `tests/test_greeter.py`. Verified the functionality via tests and manual command execution.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 14116, "tokens_out": 1686, "trimmed_requests": 0, "ts": "2026-09-27T17:41:21+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 12, "wall_s": 22.1}
{"backing": "openai/gemma-4-12b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a --shout option to the greeter. Updated `greeter/core.py` to support the `shout` parameter, `greeter/cli.py` to handle the `--shout` flag, and updated `tests/test_greeter.py` with new test cases. Verified the functionality with both tests and a manual check of the CLI.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 14068, "tokens_out": 1463, "trimmed_requests": 0, "ts": "2026-09-27T17:41:41+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 12, "wall_s": 19.5}
{"backing": "openai/gemma-4-12b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a `--shout` option to the greeter. Updated `greeter/core.py` to support the `shout` parameter, `greeter/cli.py` to include the `--shout` flag, and added corresponding tests in `tests/test_greeter.py`. Verified that `python3 -m greeter --shout Bob` prints `HELLO, BOB!`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 19838, "tokens_out": 4424, "trimmed_requests": 0, "ts": "2026-09-27T17:42:35+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 15, "wall_s": 53.7}
{"backing": "openai/gemma-4-12b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a --shout option to the greeter. Updated `greeter/core.py` to support the `shout` parameter, `greeter/cli.py` to include the `--shout` flag, and added tests to `tests/test_greeter.py`. Verified that the greeting is correctly shouted in uppercase when the flag is provided.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 12000, "tokens_out": 1523, "trimmed_requests": 0, "ts": "2026-09-27T17:42:55+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 11, "wall_s": 20.0}
{"backing": "openai/gemma-4-12b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added the `--shout` option to the greeter. Updated `greeter/core.py` to support shouting, `greeter/cli.py` to include the CLI flag, and added corresponding tests in `tests/test_greeter.py`. Verified the changes with tests and manual command execution.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 14049, "tokens_out": 1697, "trimmed_requests": 0, "ts": "2026-09-27T17:43:17+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 12, "wall_s": 22.1}
{"backing": "openai/gemma-4-12b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created `slugify.py` with the `slugify` function, created `test_slugify.py` with passing tests, and verified everything.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3283, "tokens_out": 451, "trimmed_requests": 0, "ts": "2026-09-27T17:43:24+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 6.3}
{"backing": "openai/gemma-4-12b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created slugify.py with the slugify function, created test_slugify.py with pytest tests, and verified all tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3259, "tokens_out": 458, "trimmed_requests": 0, "ts": "2026-09-27T17:43:31+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 7.0}
{"backing": "openai/gemma-4-12b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created slugify.py with the slugify function, created test_slugify.py with required test cases, and verified that tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3215, "tokens_out": 672, "trimmed_requests": 0, "ts": "2026-09-27T17:43:41+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 9.1}
{"backing": "openai/gemma-4-12b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "I created `slugify.py` with the `slugify` function, added `test_slugify.py` with the required test cases, and verified that all tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3219, "tokens_out": 539, "trimmed_requests": 0, "ts": "2026-09-27T17:43:48+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 7.7}
{"backing": "openai/gemma-4-12b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-12b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created slugify.py with the slugify function, added test_slugify.py, and verified that all tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3223, "tokens_out": 629, "trimmed_requests": 0, "ts": "2026-09-27T17:43:57+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 8.7}
bench/results/2026-09-27-gemma-4-26b-a4b.jsonl
+20
View File
@@ -0,0 +1,20 @@
{"backing": "openai/gemma-4-26b-a4b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed `total_value` in `inventory.py` to multiply `unit_price` by `quantity`. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 4904, "tokens_out": 372, "trimmed_requests": 0, "ts": "2026-09-27T17:53:38+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 4.1}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly multiply each item's `unit_price` by its `quantity`. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 4919, "tokens_out": 320, "trimmed_requests": 0, "ts": "2026-09-27T17:53:42+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 4.1}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly multiply each item's unit price by its quantity. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 4924, "tokens_out": 320, "trimmed_requests": 0, "ts": "2026-09-27T17:53:47+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 4.1}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to multiply `unit_price` by `quantity` for each item, and verified with tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 4909, "tokens_out": 410, "trimmed_requests": 0, "ts": "2026-09-27T17:53:52+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 4.8}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed `total_value` in `inventory.py` to multiply `unit_price` by `quantity`. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 4909, "tokens_out": 401, "trimmed_requests": 0, "ts": "2026-09-27T17:53:57+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 4.7}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect rate for region code R237 in rates.py. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 26136, "tokens_out": 4367, "trimmed_requests": 0, "ts": "2026-09-27T17:54:34+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 36.6}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect tax rate for region code R237 in `rates.py` from 0.21 to 0.12, which was (237 mod 25) / 100. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 26136, "tokens_out": 4494, "trimmed_requests": 0, "ts": "2026-09-27T17:55:12+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 37.5}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect tax rate for region R237 in rates.py. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 26118, "tokens_out": 4464, "trimmed_requests": 0, "ts": "2026-09-27T17:55:50+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 37.2}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect rate for region code R237 in rates.py. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 26118, "tokens_out": 4362, "trimmed_requests": 0, "ts": "2026-09-27T17:56:26+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 36.5}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py to be 0.12, which matches (237 mod 25) / 100. Verified with all tests passing.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 26118, "tokens_out": 4490, "trimmed_requests": 0, "ts": "2026-09-27T17:57:04+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 37.4}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added `--shout` option to `greeter` CLI. Updated `greet` to support shouting, added `--shout` flag to CLI, and added tests for both. All tests passed and CLI output verified.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 11250, "tokens_out": 1056, "trimmed_requests": 0, "ts": "2026-09-27T17:57:15+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 10.3}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter. \n- Updated greeter/core.py to support shout argument.\n- Updated greeter/cli.py to support --shout flag.\n- Added tests for shout functionality in tests/test_greeter.py.\n- Verified with pytest and manual execution.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 10805, "tokens_out": 1137, "trimmed_requests": 0, "ts": "2026-09-27T17:57:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 11.4}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter.\n1. Modified `greeter/core.py` to support `shout` parameter in `greet`.\n2. Modified `greeter/cli.py` to add `--shout` flag and pass it to `greet`.\n3. Added tests in `tests/test_greeter.py` for both `greet` with `shout=True` and CLI with `--shout`.\n4. Verified with `pytest` and manual CLI execution.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 11222, "tokens_out": 1321, "trimmed_requests": 0, "ts": "2026-09-27T17:57:40+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 12.7}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter. \n1. Updated `greeter/core.py` to support `shout` parameter.\n2. Updated `greeter/cli.py` to include `--shout` flag.\n3. Added tests for shout functionality in `tests/test_greeter.py`.\n4. Verified with pytest and manual execution.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 11237, "tokens_out": 1249, "trimmed_requests": 0, "ts": "2026-09-27T17:57:52+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 12.2}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter CLI and implemented shouting logic in the core. Added tests for both core logic and CLI output.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 11242, "tokens_out": 1001, "trimmed_requests": 0, "ts": "2026-09-27T17:58:03+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 10.4}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify function and tests. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3308, "tokens_out": 964, "trimmed_requests": 0, "ts": "2026-09-27T17:58:12+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 9.2}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify function in slugify.py and verified it with tests in test_slugify.py.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3433, "tokens_out": 1136, "trimmed_requests": 0, "ts": "2026-09-27T17:58:22+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 9.6}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added `slugify` function in `slugify.py` and verified with tests in `test_slugify.py`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3302, "tokens_out": 479, "trimmed_requests": 0, "ts": "2026-09-27T17:58:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 4.7}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify function and corresponding tests. All tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3312, "tokens_out": 797, "trimmed_requests": 0, "ts": "2026-09-27T17:58:35+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 7.0}
{"backing": "openai/gemma-4-26b-a4b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-26b-a4b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added `slugify` function in `slugify.py` and verified it with tests in `test_slugify.py`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3293, "tokens_out": 867, "trimmed_requests": 0, "ts": "2026-09-27T17:58:43+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 7.7}
bench/results/2026-09-27-gemma-4-31b.jsonl
+20
View File
@@ -0,0 +1,20 @@
{"backing": "openai/gemma-4-31b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the bug in `total_value` in `inventory.py` where it was summing only the unit prices instead of multiplying unit price by quantity for each item. Verified the fix with `pytest`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5927, "tokens_out": 610, "trimmed_requests": 0, "ts": "2026-09-27T18:10:31+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 18.0}
{"backing": "openai/gemma-4-31b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly multiply unit price by quantity for each item. Verified the fix with pytest.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5933, "tokens_out": 810, "trimmed_requests": 0, "ts": "2026-09-27T18:10:54+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 22.5}
{"backing": "openai/gemma-4-31b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the bug in `total_value` in `inventory.py` where only the unit price was being added instead of unit price times quantity. Verified the fix by running pytest.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5939, "tokens_out": 561, "trimmed_requests": 0, "ts": "2026-09-27T18:11:11+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 16.6}
{"backing": "openai/gemma-4-31b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the bug in `inventory.py` where `total_value` only added the unit price instead of multiplying it by the quantity. Verified the fix with pytest.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5927, "tokens_out": 598, "trimmed_requests": 0, "ts": "2026-09-27T18:11:29+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 17.5}
{"backing": "openai/gemma-4-31b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the bug in `inventory.py` where `total_value` was only summing unit prices instead of multiplying each unit price by its quantity. Verified the fix with `pytest`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5933, "tokens_out": 491, "trimmed_requests": 0, "ts": "2026-09-27T18:11:44+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 15.0}
{"backing": "openai/gemma-4-31b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region R237 in rates.py, changing it from 0.21 to 0.12 to match the required formula (237 mod 25) / 100. Verified with pytest.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 32146, "tokens_out": 1025, "trimmed_requests": 0, "ts": "2026-09-27T18:12:17+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 32.4}
{"backing": "openai/gemma-4-31b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py, changing it from 0.21 to 0.12. Verified that all tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 32124, "tokens_out": 735, "trimmed_requests": 0, "ts": "2026-09-27T18:12:42+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 24.8}
{"backing": "openai/gemma-4-31b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py from 0.21 to 0.12, as required by the test suite (237 mod 25 / 100 = 12/100 = 0.12). All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 17749, "tokens_out": 489, "trimmed_requests": 0, "ts": "2026-09-27T18:12:59+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 16.8}
{"backing": "openai/gemma-4-31b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Corrected the rate for region R237 in rates.py from 0.21 to 0.12.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 17758, "tokens_out": 548, "trimmed_requests": 0, "ts": "2026-09-27T18:13:18+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 18.3}
{"backing": "openai/gemma-4-31b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect rate for region code R237 in rates.py. Changed its value from 0.21 to 0.12. Verified that all tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 40513, "tokens_out": 4946, "trimmed_requests": 0, "ts": "2026-09-27T18:15:36+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 137.5}
{"backing": "openai/gemma-4-31b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a `--shout` option to the greeter. Modified `greeter/core.py` to support shouting and `greeter/cli.py` to handle the new flag. Added tests to verify both the core logic and the CLI interface.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9821, "tokens_out": 652, "trimmed_requests": 0, "ts": "2026-09-27T18:15:56+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 20.0}
{"backing": "openai/gemma-4-31b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter. Updated core.py to support shouting, cli.py to handle the flag, and added tests to verify both the core logic and the CLI behavior.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9854, "tokens_out": 633, "trimmed_requests": 0, "ts": "2026-09-27T18:16:16+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 19.6}
{"backing": "openai/gemma-4-31b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a --shout option to the greeter, allowing greetings to be returned in upper case. Updated core logic, CLI argument parsing, and added corresponding tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 11959, "tokens_out": 1071, "trimmed_requests": 0, "ts": "2026-09-27T18:17:42+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 11, "wall_s": 85.2}
{"backing": "openai/gemma-4-31b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added `--shout` option to the greeter. Updated `greeter/core.py` to support shouting, `greeter/cli.py` to accept the flag, and added tests in `tests/test_greeter.py`. Verified with pytest and manual execution.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9821, "tokens_out": 626, "trimmed_requests": 0, "ts": "2026-09-27T18:18:02+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 19.8}
{"backing": "openai/gemma-4-31b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added `--shout` option to greeter. Updated `greeter/core.py` to handle the `shout` argument and `greeter/cli.py` to accept the flag. Added tests to `tests/test_greeter.py` and verified they pass and that the CLI works as expected.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9809, "tokens_out": 705, "trimmed_requests": 0, "ts": "2026-09-27T18:18:24+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 21.4}
{"backing": "openai/gemma-4-31b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Implemented slugify function in slugify.py and verified it with pytest tests in test_slugify.py.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3358, "tokens_out": 348, "trimmed_requests": 0, "ts": "2026-09-27T18:18:34+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 10.5}
{"backing": "openai/gemma-4-31b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a `slugify` function in `slugify.py` and verified it with tests in `test_slugify.py`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3242, "tokens_out": 438, "trimmed_requests": 0, "ts": "2026-09-27T18:18:47+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 12.3}
{"backing": "openai/gemma-4-31b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created slugify.py with the requested slugify function and verified it with pytest tests in test_slugify.py.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3234, "tokens_out": 493, "trimmed_requests": 0, "ts": "2026-09-27T18:19:01+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 13.6}
{"backing": "openai/gemma-4-31b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Implemented a `slugify` function in `slugify.py` that lowercases text, replaces non-alphanumeric sequences with a single hyphen, and strips leading/trailing hyphens. Added and passed tests in `test_slugify.py`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3234, "tokens_out": 461, "trimmed_requests": 0, "ts": "2026-09-27T18:19:14+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 12.8}
{"backing": "openai/gemma-4-31b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "gemma-4-31b", "model_info": {"max_input_tokens": 49152, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, q4_0, q8_0 KV, 48k, temp 1.0 top-p 0.95 top-k 64, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Implemented a `slugify` function in `slugify.py` and verified it with tests in `test_slugify.py`. All four example cases passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3298, "tokens_out": 429, "trimmed_requests": 0, "ts": "2026-09-27T18:19:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 12.2}
bench/results/2026-09-27-granite-4.2-8b.jsonl
+40
View File
@@ -0,0 +1,40 @@
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply unit_price by quantity for each item, passing all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 7709, "tokens_out": 868, "trimmed_requests": 0, "ts": "2026-09-27T17:17:23+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 9.3}
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply unit_price by quantity for each item, making tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6196, "tokens_out": 681, "trimmed_requests": 0, "ts": "2026-09-27T17:17:31+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 7.5}
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply unit_price by quantity for each item, making total_value compute correctly. All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6358, "tokens_out": 752, "trimmed_requests": 0, "ts": "2026-09-27T17:17:39+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 8.1}
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed `total_value` function in `inventory.py` to multiply unit price by quantity for each item, making the total value correct. Tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6335, "tokens_out": 881, "trimmed_requests": 0, "ts": "2026-09-27T17:17:49+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 9.3}
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply unit_price by quantity for each item, passing all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 7614, "tokens_out": 721, "trimmed_requests": 0, "ts": "2026-09-27T17:17:57+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 7.9}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py from 0.21 to 0.12 (since 237 mod 25 = 12, 12/100 = 0.12). All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 23909, "tokens_out": 2327, "trimmed_requests": 0, "ts": "2026-09-27T17:18:23+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 25.8}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 from 0.21 to 0.12 (since 237 mod 25 = 12, i.e., 0.12). All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 33528, "tokens_out": 4836, "trimmed_requests": 0, "ts": "2026-09-27T17:19:16+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 52.4}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed the single wrong rate entry for region code R237. Changed from 0.21 to 0.12 (since 237 mod 25 = 12, 12/100 = 0.12). All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 31300, "tokens_out": 4765, "trimmed_requests": 0, "ts": "2026-09-27T17:20:08+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 51.3}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed the one wrong entry in the rate table: region code R237 had value 0.21, should be (237 mod 25) / 100 = 12 / 100 = 0.12. Changed the value in rates.py accordingly. All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 33528, "tokens_out": 4596, "trimmed_requests": 0, "ts": "2026-09-27T17:20:59+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 50.2}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py. The correct rate should be (237 mod 25)/100 = 12/100 = 0.12, but it was incorrectly set to 0.21. Changed the value from 0.21 to 0.12. All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 23254, "tokens_out": 4940, "trimmed_requests": 0, "ts": "2026-09-27T17:21:52+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 53.0}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Added --shout option to greeter: updated greet in core.py to accept shout=False, upper-cased output when True; added --shout flag in cli.py; added two tests (one for greet with shout=True, one for CLI with --shout). All tests pass and CLI works as expected.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 18519, "tokens_out": 1737, "trimmed_requests": 0, "ts": "2026-09-27T17:22:11+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 12, "wall_s": 18.2}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Added --shout option to greeter. Modified core.greet to accept shout=False and return uppercase when True. Modified cli.py to add --shout flag and pass it to greet. Added two tests (one for core.greet with shout=True and one for CLI with --shout). All tests pass and CLI works correctly.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 20265, "tokens_out": 2239, "trimmed_requests": 0, "ts": "2026-09-27T17:22:34+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 13, "wall_s": 22.8}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Added --shout option to greeter: updated core.greet to accept shout=False and uppercase the name when True; updated cli.py to add --shout flag and pass it to greet; added two tests for shout behavior (one for core.greet, one for CLI via capsys); all tests pass and CLI prints 'HELLO, BOB!' when using --shout.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 25055, "tokens_out": 2097, "trimmed_requests": 0, "ts": "2026-09-27T17:22:56+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 14, "wall_s": 21.9}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Added --shout option to greeter: core.greet now takes shout=False; cli adds --shout flag; tests added for both core and cli shout behavior; all tests pass and command works.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 18240, "tokens_out": 2766, "trimmed_requests": 0, "ts": "2026-09-27T17:23:24+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 12, "wall_s": 27.8}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Successfully added --shout option to greeter. Modified core.py to accept shout=bool and return uppercase when True. Modified cli.py to add --shout flag and pass it to greet. Added two tests for greet(shout=True) and main([\"--shout\", \"Bob\"]). All tests pass and the CLI works as expected.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 21491, "tokens_out": 2173, "trimmed_requests": 0, "ts": "2026-09-27T17:23:47+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 13, "wall_s": 22.2}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Created slugify.py with slugify function that lowercases text, replaces runs of non-alnum with a single hyphen, and strips hyphens from both ends. Created test_slugify.py with pytest tests covering the four required examples and additional edge cases. All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 10388, "tokens_out": 4091, "trimmed_requests": 0, "ts": "2026-09-27T17:24:28+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 40.5}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Added slugify.py with slugify(text: str) -> str that lowercases, replaces non-alphanumeric runs with single hyphens, and strips hyphens from both ends. Created test_slugify.py with pytest tests covering the four examples, all passing.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 5479, "tokens_out": 2510, "trimmed_requests": 0, "ts": "2026-09-27T17:24:53+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 24.8}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Created slugify.py with slugify function that lowercases, replaces runs of non-alphanumeric with hyphens, and strips hyphens from ends. Created test_slugify.py with four tests covering the examples. All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 5576, "tokens_out": 1993, "trimmed_requests": 0, "ts": "2026-09-27T17:25:13+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 19.8}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Added slugify function in slugify.py that lowercases text, replaces non-alphanumeric runs with a single hyphen, and strips hyphens from ends. Created test_slugify.py with four tests covering examples, all passing.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 5642, "tokens_out": 1941, "trimmed_requests": 0, "ts": "2026-09-27T17:25:32+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 19.3}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090", "outcome": "completed", "outcome_note": "Added slugify function in slugify.py that lowercases text, replaces runs of non-alphanumeric characters with a single hyphen, and strips hyphens from ends. Created test_slugify.py with pytest tests covering the four examples; all tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 5677, "tokens_out": 2688, "trimmed_requests": 0, "ts": "2026-09-27T17:25:59+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 26.6}
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value function in inventory.py by multiplying unit_price by quantity for each item. Tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6191, "tokens_out": 532, "trimmed_requests": 0, "ts": "2026-09-27T18:20:29+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 6.1}
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value function in inventory.py to multiply unit_price by quantity for each item, making it sum unit_price * quantity instead of just unit_price. Tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6191, "tokens_out": 776, "trimmed_requests": 0, "ts": "2026-09-27T18:20:38+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 8.4}
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value function in inventory.py to multiply unit_price by quantity for each item, then summed. Both tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6191, "tokens_out": 539, "trimmed_requests": 0, "ts": "2026-09-27T18:20:44+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 6.1}
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value function to multiply unit_price by quantity for each item, matching the docstring and test expectation. All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6186, "tokens_out": 728, "trimmed_requests": 0, "ts": "2026-09-27T18:20:53+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 7.9}
{"backing": "openai/granite-4.2-8b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply unit_price by quantity for each item, matching test expectation. All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 7648, "tokens_out": 629, "trimmed_requests": 0, "ts": "2026-09-27T18:21:00+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 7.1}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 from 0.21 to 0.12 (since 237 mod 25 = 12). All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 23270, "tokens_out": 4031, "trimmed_requests": 0, "ts": "2026-09-27T18:21:44+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 43.6}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 to (237 mod 25)/100 = 12/100 = 0.12.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 16630, "tokens_out": 1453, "trimmed_requests": 0, "ts": "2026-09-27T18:22:01+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 16.6}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the single wrong rate entry for region code R237 in rates.py. Changed value from 0.21 to 0.12 (since 237 mod 25 = 12). All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 23270, "tokens_out": 4556, "trimmed_requests": 0, "ts": "2026-09-27T18:22:51+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 49.2}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py from 0.21 to 0.12 (since 237 mod 25 = 12, 12/100 = 0.12). All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 16735, "tokens_out": 1593, "trimmed_requests": 0, "ts": "2026-09-27T18:23:09+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 18.0}
{"backing": "openai/granite-4.2-8b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 from 0.21 to 0.12 (since 237 mod 25 = 12, divided by 100). All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 23188, "tokens_out": 5076, "trimmed_requests": 0, "ts": "2026-09-27T18:24:05+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 55.1}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter: core.greet now accepts shout=bool=False and returns uppercase when True; cli.py adds --shout store_true flag and passes it to greet; tests added for both greet(shout=True) and main(['--shout', 'Bob']). All tests pass and CLI prints 'HELLO, BOB!' when --shout is used.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 24779, "tokens_out": 2576, "trimmed_requests": 0, "ts": "2026-09-27T18:24:32+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 14, "wall_s": 26.9}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter. Modified core.greet to accept shout=False and return uppercase when True. Added --shout flag to CLI and passed it to greet. Added two tests: one for greet with shout=True and one for CLI with --shout flag. All tests pass and CLI output matches expected.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 14785, "tokens_out": 3312, "trimmed_requests": 0, "ts": "2026-09-27T18:25:05+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 33.1}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter: core.greet now accepts shout=bool default False and returns upper case when True; cli.py adds --shout flag and passes it; tests added for both greet and CLI; all tests pass and command prints correctly.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 18591, "tokens_out": 1921, "trimmed_requests": 0, "ts": "2026-09-27T18:25:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 12, "wall_s": 19.8}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.05s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter. Updated greet function in core.py with shout parameter; updated cli.py to add --shout flag and pass it to greet; added two tests (one for greet with shout=True and one for CLI with --shout). All tests pass and CLI prints 'HELLO, BOB!' when invoked with --shout.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 18584, "tokens_out": 1434, "trimmed_requests": 0, "ts": "2026-09-27T18:25:41+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 12, "wall_s": 15.2}
{"backing": "openai/granite-4.2-8b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter: core.greet now takes shout=False keyword, cli adds --shout flag, tests added for both core and CLI, all tests pass and CLI works correctly.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 20462, "tokens_out": 1703, "trimmed_requests": 0, "ts": "2026-09-27T18:25:59+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 13, "wall_s": 17.8}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Successfully implemented slugify function in slugify.py that lowercases text, replaces every run of non-alphanumeric characters (a-z, 0-9) with a single hyphen, and strips hyphens from both ends. Created test_slugify.py with pytest tests for the four required examples and ran tests; all pass. Verified edge cases.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 30463, "tokens_out": 24495, "trimmed_requests": 0, "ts": "2026-09-27T18:30:05+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 14, "wall_s": 245.6}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify function in slugify.py using regex substitution to replace non-alnum characters with hyphens and strip hyphens from ends. Created test_slugify.py with four tests covering the examples. All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 8921, "tokens_out": 6467, "trimmed_requests": 0, "ts": "2026-09-27T18:31:09+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 63.9}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created slugify.py with the required function, test_slugify.py with pytest tests for the four examples, and all tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 9089, "tokens_out": 3303, "trimmed_requests": 0, "ts": "2026-09-27T18:31:42+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 33.1}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created slugify.py with slugify function that lowercases text, replaces runs of non-alphanumeric characters with a single hyphen, and strips hyphens from both ends. Created test_slugify.py with pytest tests for the four examples. All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 5948, "tokens_out": 1905, "trimmed_requests": 0, "ts": "2026-09-27T18:32:02+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 19.0}
{"backing": "openai/granite-4.2-8b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "granite-4.2-8b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 128k, temp 1.0 top-p 0.95, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify function in slugify.py that lowercases text, replaces runs of non-alnum characters with a single hyphen, and strips hyphens from both ends. Created test_slugify.py with four tests covering the examples, and all tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 5551, "tokens_out": 2521, "trimmed_requests": 0, "ts": "2026-09-27T18:32:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 25.0}
bench/results/2026-09-27-local-fast.jsonl
+20
View File
@@ -0,0 +1,20 @@
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1578, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-27T18:35:22+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 1.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1580, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-27T18:35:23+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 1.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1576, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-27T18:35:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 1.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1582, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-27T18:35:26+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 1.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "29.0 \u00b1 2.9e-05\nE \nE comparison failed\nE Obtained: 0.35\nE Expected: 29.0 \u00b1 2.9e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n________________________ test_total_value[items2-37.5] _________________________\n\nitems = [Item(name='gear', unit_price=12.5, quantity=3)], expected = 37.5\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 12.5 == 37.5 \u00b1 3.7e-05\nE \nE comparison failed\nE Obtained: 12.5\nE Expected: 37.5 \u00b1 3.7e-05\n\n.bench_check/test_hidden.py:22: AssertionError\n_________________________ test_total_value[items3-0.0] _________________________\n\nitems = [Item(name='empty', unit_price=9.99, quantity=0)], expected = 0.0\n\n @pytest.mark.parametrize(\n (\"items\", \"expected\"),\n [\n ([], 0.0),\n ([Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)], 29.0),\n ([Item(\"gear\", 12.5, 3)], 37.5),\n ([Item(\"empty\", 9.99, 0)], 0.0),\n ],\n )\n def test_total_value(items: list[Item], expected: float) -> None:\n> assert total_value(items) == pytest.approx(expected)\nE assert 9.99 == 0.0 \u00b1 1.0e-12\nE \nE comparison failed\nE Obtained: 9.99\nE Expected: 0.0 \u00b1 1.0e-12\n\n.bench_check/test_hidden.py:22: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_total_value[items1-29.0] - assert 0....\nFAILED .bench_check/test_hidden.py::test_total_value[items2-37.5] - assert 12...\nFAILED .bench_check/test_hidden.py::test_total_value[items3-0.0] - assert 9.9...\n3 failed, 3 passed in 0.03s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF. [100%]\n=================================== FAILURES ===================================\n______________________ test_total_value_counts_every_unit ______________________\n\n def test_total_value_counts_every_unit() -> None:\n items = [Item(\"bolt\", 0.25, 100), Item(\"nut\", 0.10, 40)]\n> assert total_value(items) == 29.0\nE AssertionError: assert 0.35 =", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 1578, "tokens_out": 50, "trimmed_requests": 0, "ts": "2026-09-27T18:35:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 1.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n......................F................................................. [ 95%]\n............. [100%]\n=================================== FAILURES ===================================\n________________________________ test_rate[237] ________________________________\n\nn = 237\n\n @pytest.mark.parametrize(\"n\", range(300))\n def test_rate(n: int) -> None:\n> assert RATES[f\"R{n:03d}\"] == pytest.approx(n % 25 / 100)\nE assert 0.21 == 0.12 \u00b1 1.2e-07\nE \nE comparison failed\nE Obtained: 0.21\nE Expected: 0.12 \u00b1 1.2e-07\n\n.bench_check/test_hidden.py:11: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_rate[237] - assert 0.21 == 0.12 \u00b1 1....\n1 failed, 300 passed in 0.21s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n........................................................................ [ 24%]\n........................................................................ [ 48%]\n........................................................................ [ 72%]\n.....................F.................................................. [ 96%]\n............ [100%]\n=================================== FAILU", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 20975, "tokens_out": 4339, "trimmed_requests": 0, "ts": "2026-09-27T18:36:02+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 34.3}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmplpnklvun/work/rates.py\", line 1\nE \"R237\": 0.12,\nE IndentationError: unexpected indent\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.12s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 10987, "tokens_out": 183, "trimmed_requests": 0, "ts": "2026-09-27T18:36:06+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 3.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmp240rbprt/work/rates.py\", line 1\nE \"R237\": 0.12,\nE IndentationError: unexpected indent\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.12s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 10987, "tokens_out": 183, "trimmed_requests": 0, "ts": "2026-09-27T18:36:09+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 3.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmpagzonvyn/work/rates.py\", line 1\nE \"R237\": 0.12,\nE IndentationError: unexpected indent\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.12s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 10974, "tokens_out": 183, "trimmed_requests": 0, "ts": "2026-09-27T18:36:12+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 3.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmp9kctkv9f/work/rates.py\", line 1\nE \"R237\": 0.12,\nE IndentationError: unexpected indent\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.12s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 10976, "tokens_out": 181, "trimmed_requests": 0, "ts": "2026-09-27T18:36:16+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 3.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter. Tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 10766, "tokens_out": 549, "trimmed_requests": 0, "ts": "2026-09-27T18:36:21+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 4.7}
{"backing": "openai/qwen3-8b-awq", "check_output": "F.FF. [100%]\n=================================== FAILURES ===================================\n__________________________ test_default_is_unchanged ___________________________\n\n def test_default_is_unchanged() -> None:\n> assert greet(\"Bob\") == \"Hello, Bob!\"\nE AssertionError: assert 'HELLO, Bob!' == 'Hello, Bob!'\nE \nE - Hello, Bob!\nE + HELLO, Bob!\n\n.bench_check/test_hidden.py:9: AssertionError\n_______________________________ test_cli_default _______________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x7b6d8abfca70>\n\n def test_cli_default(capsys) -> None:\n> assert main([\"Ada\"]) == 0\nE AssertionError: assert None == 0\nE + where None = main(['Ada'])\n\n.bench_check/test_hidden.py:17: AssertionError\n----------------------------- Captured stdout call -----------------------------\nHELLO, Ada!\n________________________________ test_cli_shout ________________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x7b6d8abbfb30>\n\n def test_cli_shout(capsys) -> None:\n> assert main([\"--shout\", \"Ada\"]) == 0\nE AssertionError: assert None == 0\nE + where None = main(['--shout', 'Ada'])\n\n.bench_check/test_hidden.py:22: AssertionError\n----------------------------- Captured stdout call -----------------------------\nHELLO, ADA!\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_default_is_unchanged - AssertionErro...\nFAILED .bench_check/test_hidden.py::test_cli_default - AssertionError: assert...\nFAILED .bench_check/test_hidden.py::test_cli_shout - AssertionError: assert N...\n3 failed, 2 passed in 0.05s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter. Tests pass.", "prose_turns": 0, "provider": "openai", "success": false, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 6758, "tokens_out": 474, "trimmed_requests": 0, "ts": "2026-09-27T18:36:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 4.0}
{"backing": "openai/qwen3-8b-awq", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter. Tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 10775, "tokens_out": 549, "trimmed_requests": 0, "ts": "2026-09-27T18:36:30+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 4.7}
{"backing": "openai/qwen3-8b-awq", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to greeter. Tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 10775, "tokens_out": 549, "trimmed_requests": 0, "ts": "2026-09-27T18:36:35+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 4.7}
{"backing": "openai/qwen3-8b-awq", "check_output": "t('stderr') is not None:\n raise ValueError('stdout and stderr arguments may not be used '\n 'with capture_output.')\n kwargs['stdout'] = PIPE\n kwargs['stderr'] = PIPE\n \n with Popen(*popenargs, **kwargs) as process:\n try:\n stdout, stderr = process.communicate(input, timeout=timeout)\n except TimeoutExpired as exc:\n process.kill()\n if _mswindows:\n # Windows accumulates the output in a single blocking\n # read() call run on child threads, with the timeout\n # being done in a join() on those threads. communicate()\n # _after_ kill() is required to collect that and add it\n # to the exception.\n exc.stdout, exc.stderr = process.communicate()\n else:\n # POSIX _communicate already populated the output so\n # far into the TimeoutExpired exception.\n process.wait()\n raise\n except: # Including KeyboardInterrupt, communicate handled that.\n process.kill()\n # We don't call process.wait() as .__exit__ does that for us.\n raise\n retcode = process.poll()\n if check and retcode:\n> raise CalledProcessError(retcode, process.args,\n output=stdout, stderr=stderr)\nE subprocess.CalledProcessError: Command '['/root/nisse/.venv/bin/python', '-m', 'greeter', '--shout', 'Bob']' returned non-zero exit status 2.\n\n/usr/lib/python3.12/subprocess.py:571: CalledProcessError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_cli_shout - SystemExit: 2\nFAILED .bench_check/test_hidden.py::test_module_entry_point - subprocess.Call...\n2 failed, 3 passed in 0.13s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "executor call failed: upstream returned 429: {\"error\":{\"message\":\"Rate limit exceeded for api_key: 8941a8ca890cd15add31e5bcb3204db273d130b7f62dede0557b3a11c1a2b52b. Limit type: requests. Current limit: 60, Remaining: 0. Limit resets at: 2026-09-27 18:37:56 UTC\",\"type\":\"throttling_error\",\"param\":null,\"code\":\"429\"}}", "prose_turns": 0, "provider": "openai", "success": false, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 1616, "tokens_out": 97, "trimmed_requests": 0, "ts": "2026-09-27T18:36:57+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 20.9}
{"backing": "openai/qwen3-8b-awq", "check_output": " (\"Hello, World!\", \"hello-world\"),\n (\" Nisse Agent \", \"nisse-agent\"),\n (\"a--b\", \"a-b\"),\n (\"\", \"\"),\n (\"Python 3.12 Release\", \"python-3-12-release\"),\n (\"--leading and trailing--\", \"leading-and-trailing\"),\n (\"MiXeD CaSe\", \"mixed-case\"),\n (\"!!!\", \"\"),\n ],\n )\n def test_slugify(text: str, expected: str) -> None:\n> assert slugify(text) == expected\nE AssertionError: assert '' == 'leading-and-trailing'\nE \nE - leading-and-trailing\n\n.bench_check/test_hidden.py:19: AssertionError\n_____________________ test_slugify[MiXeD CaSe-mixed-case] ______________________\n\ntext = 'MiXeD CaSe', expected = 'mixed-case'\n\n @pytest.mark.parametrize(\n (\"text\", \"expected\"),\n [\n (\"Hello, World!\", \"hello-world\"),\n (\" Nisse Agent \", \"nisse-agent\"),\n (\"a--b\", \"a-b\"),\n (\"\", \"\"),\n (\"Python 3.12 Release\", \"python-3-12-release\"),\n (\"--leading and trailing--\", \"leading-and-trailing\"),\n (\"MiXeD CaSe\", \"mixed-case\"),\n (\"!!!\", \"\"),\n ],\n )\n def test_slugify(text: str, expected: str) -> None:\n> assert slugify(text) == expected\nE AssertionError: assert '' == 'mixed-case'\nE \nE - mixed-case\n\n.bench_check/test_hidden.py:19: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_slugify[Hello, World!-hello-world]\nFAILED .bench_check/test_hidden.py::test_slugify[ Nisse Agent -nisse-agent]\nFAILED .bench_check/test_hidden.py::test_slugify[a--b-a-b] - AssertionError: ...\nFAILED .bench_check/test_hidden.py::test_slugify[Python 3.12 Release-python-3-12-release]\nFAILED .bench_check/test_hidden.py::test_slugify[--leading and trailing---leading-and-trailing]\nFAILED .bench_check/test_hidden.py::test_slugify[MiXeD CaSe-mixed-case] - Ass...\n6 failed, 2 passed in 0.03s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF [100%]\n=================================== FAILURES ===================================\n_________________________________ test_slugify _________________________________\n\n def test_slugify():\n> assert slugify('Hello, World!') == 'hello-world'\n ^^^^^^^\nE NameError: name 'slugify' is not defined\n\ntest_slugify.py:4: NameError\n===============", "prose_turns": 0, "provider": "openai", "success": false, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3658, "tokens_out": 250, "trimmed_requests": 0, "ts": "2026-09-27T18:37:19+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 22.6}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from slugify import slugify\nE File \"/tmp/tmpj_2vxo2m/work/slugify.py\", line 1\nE def slugify(text: str) -> str:\\n import re\\n text = text.lower()\\n text = re.sub(r'[^a-z0-9]+', '-', text)\\n text = text.strip('-')\\n return text\nE ^\nE SyntaxError: unexpected character after line continuation character\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.12s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n_______________________ ERROR collecting test_slugify.py _______________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 5808, "tokens_out": 331, "trimmed_requests": 0, "ts": "2026-09-27T18:37:23+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 3.4}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from slugify import slugify\nE File \"/tmp/tmpboff_f4_/work/slugify.py\", line 1\nE def slugify(text: str) -> str:\\n import re\\n text = text.lower()\\n text = re.sub(r'[^a-z0-9]+', '-', text)\\n return text.strip('-')\\n\nE ^\nE SyntaxError: unexpected character after line continuation character\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.12s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n_______________________ ERROR collecting test_slugify.py _______________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 5825, "tokens_out": 380, "trimmed_requests": 0, "ts": "2026-09-27T18:37:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 3.8}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from slugify import slugify\nE File \"/tmp/tmpypm_nshh/work/slugify.py\", line 1\nE def slugify(text: str) -> str:\\n import re\\n text = text.lower()\\n text = re.sub(r'[^a-z0-9]+', '-', text)\\n return text.strip('-')\nE ^\nE SyntaxError: unexpected character after line continuation character\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.12s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nF [100%]\n=================================== FAILURES ===================================\n_________________________________ test_slugify _________________________________\n\n def test_slugify():\n> assert slugify('Hello, World!') == 'hello-world'\n ^^^^^^^\nE NameError: name 'slugify' is not defined\n\ntest_slugify.py:5: NameError\n===============", "prose_turns": 0, "provider": "openai", "success": false, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 5692, "tokens_out": 337, "trimmed_requests": 0, "ts": "2026-09-27T18:37:31+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 3.4}
{"backing": "openai/qwen3-8b-awq", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from slugify import slugify\nE File \"/tmp/tmp9oyadssz/work/slugify.py\", line 1\nE def slugify(text: str) -> str:\\n import re\\n text = text.lower()\\n text = re.sub(r'[^a-z0-9]+', '-', text)\\n text = text.strip('-')\\n return text\nE ^\nE SyntaxError: unexpected character after line continuation character\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.13s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-fast", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": true}, "note": "vLLM 0.30.0, AWQ, fp8 KV, 32k, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n_______________________ ERROR collecting test_slugify.py _______________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 4136, "tokens_out": 269, "trimmed_requests": 0, "ts": "2026-09-27T18:37:35+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 3.0}
bench/results/2026-09-27-mimo-v2.6-9b.jsonl
+20
View File
@@ -0,0 +1,20 @@
{"backing": "openai/mimo-v2.6-9b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply unit_price by quantity per item; all tests now pass (2 passed).", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5132, "tokens_out": 310, "trimmed_requests": 0, "ts": "2026-09-27T17:44:29+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 4.2}
{"backing": "openai/mimo-v2.6-9b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply unit_price by quantity per item. All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6915, "tokens_out": 645, "trimmed_requests": 0, "ts": "2026-09-27T17:44:37+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 8.0}
{"backing": "openai/mimo-v2.6-9b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply unit_price by quantity per item; all tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5132, "tokens_out": 306, "trimmed_requests": 0, "ts": "2026-09-27T17:44:42+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 4.3}
{"backing": "openai/mimo-v2.6-9b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply unit_price by quantity for each item; both tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5132, "tokens_out": 308, "trimmed_requests": 0, "ts": "2026-09-27T17:44:47+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 4.5}
{"backing": "openai/mimo-v2.6-9b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value() in inventory.py to multiply unit price by quantity instead of just summing unit prices. Both tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5316, "tokens_out": 414, "trimmed_requests": 0, "ts": "2026-09-27T17:44:53+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 5.6}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.21s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the R237 entry in rates.py (0.21 -> 0.12) and verified all 300 tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 12453, "tokens_out": 202, "trimmed_requests": 0, "ts": "2026-09-27T17:44:57+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 4.2}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.21s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the wrong rate for region R237 (was 0.21, now 0.12 = 237 mod 25 / 100) in rates.py. All 300 tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 25884, "tokens_out": 4404, "trimmed_requests": 0, "ts": "2026-09-27T17:45:43+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 45.4}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the one wrong rate entry in rates.py: R237 was 0.21, corrected to 0.12 (237 mod 25 = 12, /100). All 300 tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 12475, "tokens_out": 230, "trimmed_requests": 0, "ts": "2026-09-27T17:45:48+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 4.7}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the single wrong rate entry in rates.py: R237 was 0.21 instead of (237 mod 25)/100 = 0.12. All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 25884, "tokens_out": 4403, "trimmed_requests": 0, "ts": "2026-09-27T17:46:34+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 45.7}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.21s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the wrong rate for R237 in rates.py (was 0.21, now 0.12 = 237 mod 25 / 100). All 300 tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 25868, "tokens_out": 4400, "trimmed_requests": 0, "ts": "2026-09-27T17:47:20+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 45.5}
{"backing": "openai/mimo-v2.6-9b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a --shout option to the greeter: greet() takes shout: bool = False and uppercases the greeting when true; cli.py gained a --shout store_true flag passed to greet; tests added for both shout=True and the CLI with capsys. All 4 tests pass and `python -m greeter --shout Bob` prints HELLO, BOB!.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 8490, "tokens_out": 592, "trimmed_requests": 0, "ts": "2026-09-27T17:47:28+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 7.6}
{"backing": "openai/mimo-v2.6-9b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a --shout option to the greeter.\n\nChanges:\n- `greeter/core.py`: `greet` now takes `shout: bool = False`; when true it returns the greeting in uppercase.\n- `greeter/cli.py`: added a `--shout` `store_true` flag and passes it to `greet`.\n- `tests/test_greeter.py`: added `test_greet_shout` and `test_cli_prints_the_greeting_shout`.\n\nVerification: `python -m pytest -q` passes (4 passed), and `python -m greeter --shout Bob` prints `HELLO, BOB!`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 10204, "tokens_out": 809, "trimmed_requests": 0, "ts": "2026-09-27T17:47:39+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 10.0}
{"backing": "openai/mimo-v2.6-9b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a --shout option to the greeter: greet() now accepts shout=True (returns uppercase), the CLI gained a --shout store_true flag, and two tests were added (greet shout + CLI capsys). All 4 tests pass and `python -m greeter --shout Bob` prints HELLO, BOB!.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 8490, "tokens_out": 585, "trimmed_requests": 0, "ts": "2026-09-27T17:47:46+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 7.3}
{"backing": "openai/mimo-v2.6-9b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a --shout option to the greeter: greet() now accepts shout=True (returns uppercase greeting), the CLI has a --shout flag passed through, and two new tests cover both. All 4 tests pass and `python -m greeter --shout Bob` prints HELLO, BOB!.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 10964, "tokens_out": 708, "trimmed_requests": 0, "ts": "2026-09-27T17:47:56+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 9.0}
{"backing": "openai/mimo-v2.6-9b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a --shout option to the greeter.\n\n1. `greeter/core.py`: `greet` now takes `shout: bool = False`; when true it returns the greeting in upper case (`greet(\"Bob\", shout=True) == \"HELLO, BOB!\"`). Default behaviour unchanged.\n2. `greeter/cli.py`: added `parser.add_argument(\"--shout\", action=\"store_true\")` and passes `shout=args.shout` to `greet`.\n3. `tests/test_greeter.py`: added `test_greet_shout` and `test_cli_prints_the_greeting_shout` (uses capsys).\n4. `python -m pytest -q` \u2192 4 passed; `pyt", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 10198, "tokens_out": 848, "trimmed_requests": 0, "ts": "2026-09-27T17:48:06+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 10.2}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify.py with slugify() and test_slugify.py covering the four specified examples; all 4 tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 2717, "tokens_out": 324, "trimmed_requests": 0, "ts": "2026-09-27T17:48:11+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 4.3}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify.py with slugify() and test_slugify.py covering all four examples; all 4 tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 2651, "tokens_out": 277, "trimmed_requests": 0, "ts": "2026-09-27T17:48:15+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 3.7}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify() in slugify.py using a regex to collapse non-alphanumeric runs into single hyphens, plus test_slugify.py covering all four examples; all 4 tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 7110, "tokens_out": 552, "trimmed_requests": 0, "ts": "2026-09-27T17:48:22+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 7.0}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify.py with a slugify() function and test_slugify.py covering the four required examples; pytest passes (4 passed).", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 2821, "tokens_out": 384, "trimmed_requests": 0, "ts": "2026-09-27T17:48:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 4.9}
{"backing": "openai/mimo-v2.6-9b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "mimo-v2.6-9b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q6_K, q8_0 KV, 256k, temp 0.6 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify.py with slugify() and test_slugify.py covering the four specified examples; all 4 pytest tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 2808, "tokens_out": 354, "trimmed_requests": 0, "ts": "2026-09-27T17:48:32+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 4.4}
bench/results/2026-09-27-qwen3.6-35b-a3b.jsonl
+20
View File
@@ -0,0 +1,20 @@
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed `total_value` in inventory.py: changed `total += item.unit_price` to `total += item.unit_price * item.quantity` so each item contributes its unit price times its quantity to the total inventory value. All 2 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6995, "tokens_out": 651, "trimmed_requests": 0, "ts": "2026-09-27T17:49:45+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 5.3}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly multiply each item's `unit_price` by its `quantity` when accumulating the total, instead of just adding `unit_price`. Both tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6728, "tokens_out": 630, "trimmed_requests": 0, "ts": "2026-09-27T17:49:50+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 5.3}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed `total_value` in inventory.py: the bug was that it only summed `item.unit_price` without multiplying by `item.quantity`. Changed `total += item.unit_price` to `total += item.unit_price * item.quantity`. All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5219, "tokens_out": 642, "trimmed_requests": 0, "ts": "2026-09-27T17:49:56+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 5.2}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed `total_value` in `inventory.py` \u2014 changed `total += item.unit_price` to `total += item.unit_price * item.quantity` so each item contributes its unit price times its quantity. All 2 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6808, "tokens_out": 674, "trimmed_requests": 0, "ts": "2026-09-27T17:50:02+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 5.6}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` so that each item contributes `unit_price * quantity` to the total, rather than just `unit_price`. All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6285, "tokens_out": 547, "trimmed_requests": 0, "ts": "2026-09-27T17:50:07+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 5.0}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the wrong rate entry for region R237 in rates.py: changed from 0.21 to 0.12 (237 mod 25 = 12, 12/100 = 0.12). All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 18886, "tokens_out": 580, "trimmed_requests": 0, "ts": "2026-09-27T17:50:13+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 6.0}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the wrong rate for region code R237 in rates.py, changing it from 0.21 to 0.12 (since 237 mod 25 = 12, and 12/100 = 0.12). All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 18904, "tokens_out": 625, "trimmed_requests": 0, "ts": "2026-09-27T17:50:20+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 6.4}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect rate for region R237 in rates.py: changed the value from 0.21 to 0.12 (since 237 mod 25 = 12, and the rate should be (NNN mod 25) / 100 = 0.12). All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 21349, "tokens_out": 718, "trimmed_requests": 0, "ts": "2026-09-27T17:50:28+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 7.2}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region R237 in rates.py: changed the value from 0.21 to 0.12 (237 mod 25 / 100 = 0.12). All 300 tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 18717, "tokens_out": 676, "trimmed_requests": 0, "ts": "2026-09-27T17:50:35+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 6.8}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the incorrect rate for region R237 in rates.py, changing it from 0.21 to 0.12 (since 237 mod 25 = 12, and 12/100 = 0.12). All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 27343, "tokens_out": 4956, "trimmed_requests": 0, "ts": "2026-09-27T17:51:09+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 32.9}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added shout support to the greeter:\n1. `greeter/core.py`: Added `shout: bool = False` keyword argument to `greet()`, returning the greeting in uppercase when `shout=True`.\n2. `greeter/cli.py`: Added `--shout` flag (action=\"store_true\") and passed it to `greet()`.\n3. `tests/test_greeter.py`: Added two tests \u2014 one for `greet(\"Bob\", shout=True)` and one verifying `main([\"--shout\", \"Bob\"])` prints `HELLO, BOB!` via capsys.\nAll 4 tests pass and CLI verification succeeds.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 8616, "tokens_out": 1008, "trimmed_requests": 0, "ts": "2026-09-27T17:51:17+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 7.8}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "executor kept replying in prose instead of tools", "prose_turns": 2, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9357, "tokens_out": 748, "trimmed_requests": 0, "ts": "2026-09-27T17:51:24+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 6.7}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "executor kept replying in prose instead of tools", "prose_turns": 2, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9242, "tokens_out": 623, "trimmed_requests": 0, "ts": "2026-09-27T17:51:29+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 5.4}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a `--shout` option to the greeter. Modified `greeter/core.py` to accept a `shout` keyword argument that uppercases the greeting when True. Updated `greeter/cli.py` to add a `--shout` flag and pass it to `greet`. Added two tests to `tests/test_greeter.py`: one for `greet(\"Bob\", shout=True)` and one for `main([\"--shout\", \"Bob\"])` with capsys. All 4 tests pass, and `python -m greeter --shout Bob` prints `HELLO, BOB!`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 16065, "tokens_out": 1466, "trimmed_requests": 0, "ts": "2026-09-27T17:51:41+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 11.0}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "escalate", "outcome_note": "executor kept replying in prose instead of tools", "prose_turns": 2, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9418, "tokens_out": 681, "trimmed_requests": 0, "ts": "2026-09-27T17:51:47+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 5.9}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Milestone 1 complete: created slugify.py defining slugify(text) which lowercases, replaces non-alphanumeric runs with a single hyphen, and strips leading/trailing hyphens. Created test_slugify.py with 4 pytest tests covering all required examples, all passing.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 4490, "tokens_out": 538, "trimmed_requests": 0, "ts": "2026-09-27T17:51:52+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 4.5}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created `slugify.py` with a `slugify(text: str) -> str` function that lowercases text, replaces non-alphanumeric character runs with a single hyphen, and strips leading/trailing hyphens. Created `test_slugify.py` with 4 pytest tests covering the required examples. All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 4538, "tokens_out": 631, "trimmed_requests": 0, "ts": "2026-09-27T17:51:57+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 5.1}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Milestone 1 complete: created slugify.py with a slugify(text) function that lowercases text, replaces non alphanumeric runs with a single hyphen, and strips leading/trailing hyphens. Created test_slugify.py covering all four examples, and all 4 pytest tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3410, "tokens_out": 546, "trimmed_requests": 0, "ts": "2026-09-27T17:52:02+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 4.3}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created `slugify.py` with a `slugify(text)` function that lowercases text, replaces non a-z/0-9 character runs with a single hyphen, and strips leading/trailing hyphens. Created `test_slugify.py` with 4 pytest tests covering the specified examples. All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3476, "tokens_out": 613, "trimmed_requests": 0, "ts": "2026-09-27T17:52:07+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 4.7}
{"backing": "openai/qwen3.6-35b-a3b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.6-35b-a3b", "model_info": {"max_input_tokens": 262144, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q3_K_XL, q8_0 KV, 256k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Created slugify.py with a slugify() function that lowercases text, replaces non-alphanumeric runs with a single hyphen, and strips leading/trailing hyphens. Created test_slugify.py with 4 pytest tests covering the examples \"Hello, World!\" -> \"hello-world\", \" Nisse Agent \" -> \"nisse-agent\", \"a--b\" -> \"a-b\", and \"\" -> \"\". All tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3520, "tokens_out": 638, "trimmed_requests": 0, "ts": "2026-09-27T17:52:12+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 4.9}
bench/results/2026-09-27-qwen3.8-27b.jsonl
+20
View File
@@ -0,0 +1,20 @@
{"backing": "openai/qwen3.8-27b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply each item's unit price by its quantity; both tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5275, "tokens_out": 439, "trimmed_requests": 0, "ts": "2026-09-27T18:01:40+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 11.8}
{"backing": "openai/qwen3.8-27b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py to multiply each item's unit price by its quantity (was only summing unit prices). Both tests now pass (2 passed).", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6546, "tokens_out": 522, "trimmed_requests": 0, "ts": "2026-09-27T18:01:55+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 14.1}
{"backing": "openai/qwen3.8-27b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py: it now accumulates item.unit_price * item.quantity instead of just item.unit_price. All 2 tests in test_inventory.py pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6575, "tokens_out": 557, "trimmed_requests": 0, "ts": "2026-09-27T18:02:10+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 15.0}
{"backing": "openai/qwen3.8-27b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py: the loop added only item.unit_price, now it adds item.unit_price * item.quantity. Both tests pass (2 passed).", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6985, "tokens_out": 672, "trimmed_requests": 0, "ts": "2026-09-27T18:02:28+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 17.5}
{"backing": "openai/qwen3.8-27b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed total_value in inventory.py so each item contributes unit_price * quantity instead of unit_price alone. All 2 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 6575, "tokens_out": 339, "trimmed_requests": 0, "ts": "2026-09-27T18:02:38+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 10.2}
{"backing": "openai/qwen3.8-27b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the one wrong entry in rates.py: R237 changed from 0.21 to 0.12 (237 mod 25 = 12 \u2192 12/100). All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 9841, "tokens_out": 525, "trimmed_requests": 0, "ts": "2026-09-27T18:02:54+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 15.0}
{"backing": "openai/qwen3.8-27b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the one wrong entry in the rate table: `test_rate[R237]` was the only failing test, so in rates.py line 241 I changed \"R237\": 0.21 to \"R237\": 0.12 (237 mod 25 = 12 \u2192 0.12), leaving every other line untouched. All 300 tests now pass (`python -m pytest -q`).", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 13348, "tokens_out": 758, "trimmed_requests": 0, "ts": "2026-09-27T18:03:15+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 20.7}
{"backing": "openai/qwen3.8-27b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the single wrong entry in rates.py: \"R237\": 0.21 \u2192 0.12 (= 237 % 25 / 100) on line 241. All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 12292, "tokens_out": 591, "trimmed_requests": 0, "ts": "2026-09-27T18:03:31+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 15.9}
{"backing": "openai/qwen3.8-27b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Fixed the single wrong entry in rates.py: changed region code R237 from 0.21 to 0.12 (237 mod 25 / 100 = 0.12). All 300 tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 18850, "tokens_out": 548, "trimmed_requests": 0, "ts": "2026-09-27T18:03:48+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 15.9}
{"backing": "openai/qwen3.8-27b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Milestone 1 complete. Ran pytest: exactly one test failed (test_rate[R237], obtained 0.21 vs expected 0.12). Corrected line 241 of rates.py from \"R237\": 0.21 to \"R237\": 0.12 (237 % 25 / 100 = 0.12), leaving all other entries untouched. Re-ran pytest: all 300 tests pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 29415, "tokens_out": 911, "trimmed_requests": 0, "ts": "2026-09-27T18:04:13+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 24.8}
{"backing": "openai/qwen3.8-27b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter: greet() in greeter/core.py now takes shout: bool = False and upper-cases the greeting when true; greeter/cli.py adds a --shout (store_true) flag passed through to greet; two new tests added (greet shout=True, and CLI --shout via capsys). `python -m pytest -q` passes (4/4) and `python -m greeter --shout Bob` prints \"HELLO, BOB!\".", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9879, "tokens_out": 1015, "trimmed_requests": 0, "ts": "2026-09-27T18:04:37+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 24.3}
{"backing": "openai/qwen3.8-27b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added the --shout option to the greeter: (1) greeter/core.py's greet now takes a `shout: bool = False` keyword argument and returns the whole greeting in upper case when true (greet(\"Bob\", shout=True) == \"HELLO, BOB!\"), with unchanged default behaviour; (2) greeter/cli.py adds a `--shout` flag (action=\"store_true\") and passes it to greet; (3) added two tests to tests/test_greeter.py \u2014 one for greet with shout=True and one calling main([\"--shout\", \"Bob\"]) and checking printed output via capsys; (", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 17632, "tokens_out": 1868, "trimmed_requests": 0, "ts": "2026-09-27T18:05:22+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 44.0}
{"backing": "openai/qwen3.8-27b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added a --shout option to the greeter: gave greet(name, shout=False) a shout keyword argument in greeter/core.py that upper-cases the greeting, added the --shout flag (store_true) in greeter/cli.py passing it through, and added two tests (greet with shout=True, and main([\"--shout\", \"Bob\"]) with capsys). python -m pytest -q passes (4 tests) and python -m greeter --shout Bob prints HELLO, BOB!.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 8690, "tokens_out": 1116, "trimmed_requests": 0, "ts": "2026-09-27T18:05:48+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 26.5}
{"backing": "openai/qwen3.8-27b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Milestone 1 done: added a `shout: bool = False` keyword argument to `greet` in `greeter/core.py` that returns the greeting in upper case, added a `--shout` store_true flag in `greeter/cli.py` passed through to `greet`, and added two tests (greet with shout=True, and main([\"--shout\", \"Bob\"]) checked via capsys). `python -m pytest -q` shows 4 passed, and `python -m greeter --shout Bob` prints `HELLO, BOB!`.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 14842, "tokens_out": 1534, "trimmed_requests": 0, "ts": "2026-09-27T18:06:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 36.4}
{"backing": "openai/qwen3.8-27b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter per the approved plan: (1) greet(name, shout: bool = False) in greeter/core.py returns the upper-cased greeting when shout is true, default unchanged; (2) greeter/cli.py adds a --shout flag (action=\"store_true\") passed to greet; (3) tests/test_greeter.py gains test_greet_shout and test_cli_shout_prints_upper_case_greeting using capsys; (4) verified python -m pytest -q passes (4 passed) and python -m greeter --shout Bob prints HELLO, BOB! while plain python -m ", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 14751, "tokens_out": 1805, "trimmed_requests": 0, "ts": "2026-09-27T18:07:08+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 42.2}
{"backing": "openai/qwen3.8-27b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Milestone 1 complete: created `slugify.py` with a `slugify(text)` function (lowercases, collapses non-alphanumeric runs to a single hyphen, strips edge hyphens) and `test_slugify.py` with pytest tests covering all four required examples. `python -m pytest -q test_slugify.py` passes (4 passed).", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 4395, "tokens_out": 544, "trimmed_requests": 0, "ts": "2026-09-27T18:07:21+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 13.4}
{"backing": "openai/qwen3.8-27b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Milestone 1 done: created slugify.py with slugify(text) (lowercases, collapses non-a-z0-9 runs to a single hyphen, strips edge hyphens) and test_slugify.py with pytest tests for the four examples; `python -m pytest -q test_slugify.py` passes (4 passed).", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3415, "tokens_out": 557, "trimmed_requests": 0, "ts": "2026-09-27T18:07:35+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 13.3}
{"backing": "openai/qwen3.8-27b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify.py with slugify(text) (lowercase, collapse non a-z0-9 runs to single hyphen, strip hyphens), and test_slugify.py with pytest tests for all four examples. `python -m pytest -q test_slugify.py` passes: 4 passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3568, "tokens_out": 666, "trimmed_requests": 0, "ts": "2026-09-27T18:07:52+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 16.6}
{"backing": "openai/qwen3.8-27b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Added slugify.py defining slugify(text) which lowercases, replaces runs of non-[a-z0-9] with a single hyphen, and strips hyphens from both ends; added test_slugify.py with pytest tests covering all four plan examples; `python -m pytest -q test_slugify.py` passes (4 passed).", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3542, "tokens_out": 906, "trimmed_requests": 0, "ts": "2026-09-27T18:08:14+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 21.9}
{"backing": "openai/qwen3.8-27b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "8a8d8dc+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 350 W cap", "outcome": "completed", "outcome_note": "Milestone 1 done: created `slugify.py` with a `slugify` function (lowercases text, collapses non-alphanumeric runs into single hyphens, strips end hyphens) and `test_slugify.py` with pytest tests for all four required examples. `python -m pytest -q test_slugify.py` passes: 4 passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3453, "tokens_out": 501, "trimmed_requests": 0, "ts": "2026-09-27T18:08:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 13.0}
docs/Architecture-Build-Spec.md
+523 -95
View File
@@ -10,7 +10,10 @@ Gitea, Vikunja, Outline, LiteLLM) and heaviest on the **agent library** and the
document and no longer exists separately: its A-series components appear as A10–A16,
its network and credential revisions in A5 and B3, and its revised build order as
Part G. The inference-hardware question it raised has since been resolved in
principle (Part H: a 4090-class host).
principle (Part H: a 4090-class host). The MCP Deployment amendment has likewise been
folded in: the every-agent-gets-a-VM rule, per-instance egress policy, and MCP server
placement now form Part F1–F3; its consequences appear in Part 0 (site identifiers),
A7, B3, C2, C3, C5, the Part F4 Watchtower carve-out, and Part H.
---
@@ -32,6 +35,12 @@ same string across all of those: the VM, the container, the Caddy host, the Matr
account, and the Bitwarden machine account for a given agent all share one name, so
that grepping the infrastructure for `nisse-ops` finds everything.
**Sites** are identified by their domain stem: `garrett-hall` (Arlington, hosts under
`garrett-hall.com`) and `hartman-hut` (Cross Plains, hosts under `hartman-hut.com`).
As with agent names, one string is used everywhere a site appears. For an MCP server
instance (F3), that means its container name, its Vaultwarden machine account, and
its MCP tool prefix.
The three artifacts live as three packages in **one repository, `nisse`** (decision
recorded in Part H). Directory and package names carry the identity. If mirrored to
GitHub, it goes under a personal account rather than an org — `github.com/nisse` is
@@ -55,6 +64,14 @@ Both are thin applications over a shared **`nisse-core`** library. The library i
consumers that don't exist yet. Build the Dev Agent as if the library were internal,
then lift the seams that proved real.
There is one deliberate, bounded exception: the **work model's identity scheme**
(0.2) is settled for both agents ahead of extraction. Identifiers end up in the state
schema, branch names, the operational log (A9), the transcript (A10), and external
records. Changing those after extraction would mean migrating two agents and their
stored state. Only identity is designed ahead. Behaviour is still built by each
agent as it needs it, so nothing in 0.2 requires the Dev Agent to implement duties,
deduplication, or concurrency.
The three artifacts live in **one repository** (decided — Part H), so the
extraction step in the build order is a package split within one tree, not a repo
split.
@@ -77,6 +94,119 @@ split.
---
## 0.2 The Work Model: Duties, Work Items, and Scopes
Every agent's work is organised into three kinds of unit. They define what state
belongs where, how work is identified, and what the operator is resuming when they
come back to something. Components throughout this document refer to them.
| Concept | What it is | Lifetime | Dev Agent | Sysadmin Agent |
|---|---|---|---|---|
| **Duty** | A standing responsibility that produces work items | Indefinite, recurring | None yet; the operator is the only source | Scheduled checks, log monitoring, alert intake |
| **Work item** | One effort with a lifecycle: opened, worked (often through gates), closed | Opened → closed | Building a feature through arch plan → milestones → PRs | Diagnosing one failure; answering one question |
| **Scope** | What a work item operates on, and the context that persists across work items there | Indefinite | A repo | A site and service, e.g. `garrett-hall` Proxmox |
### Duty
- **A duty produces work items. It is not one itself.** A run of a duty (one log
analysis, one scheduled check) is read-only and yields findings. A finding that
needs attention becomes a work item.
- **The operator is always an implicit source** of work items, for every agent.
- **Deduplication is a duty's responsibility.** When a later run finds the same
failure, it attaches to the existing open work item instead of opening a second
one. That needs a stable fingerprint per finding, checked against open work items.
This is the same idempotency concern A2/A3 address for side effects.
### Work item
- **The unit of state.** A work item owns its state-machine position, gates and plan
revisions, pending questions, pinned role mapping (A11), and its transcript (A10).
- **Identity is minted locally.** Every work item gets a local ID when it opens.
This is the `task_id` of A9: a work item's ID and its correlation ID are one value.
- **An external record is optional, and linked.** A work item *may* be tied to a
durable external record: a Gitea issue for the Dev Agent, a Vikunja ticket for the
Sysadmin Agent. Many will not be; a general question to either agent is a work
item with no record. A work item can also gain a record partway through, for
example a question that turns into a change to make.
- **Where the spec requires a record, it has one.** Dev plan gates are externalized
to Gitea (Part H), and ticket-driven ops work has its ticket (C4). When a record is
linked, external truth wins on divergence (A2), and gate state can be re-derived
from it after state loss. Whether *every* gated work item must have a record is
open (Part H).
- **Work items are not nested.** A milestone is a phase *within* a dev work item, and
a delegation (A13) is an episode within one. Neither is a work item of its own.
### Scope
- **A scope holds context shared by the work items that operate on it:** for a repo,
the clone and the project's conventions; for infrastructure, the A17 world model
of that site or service.
- **Many work items may share one scope.** A feature waiting on review and a bug fix
in the same repo are two work items on one scope. So are two failures found in the
same log run.
- **A work item has zero or more scopes.** A dev work item has exactly one repo for
now. A general question may have none.
- **Scope identifiers follow existing naming.** Repos use `owner/repo`. Sites use the
site identifier (Part 0), combined with the service.
### Park-and-switch: one active work item per agent
- **Each agent has at most one active work item.** Others are *parked*, with their
full position held in the state store (A2).
- **Switching happens only at safe points,** where the loop is waiting on a human
(an open gate, a pending question, a PR in review) or is idle. It never happens
mid-turn. Because turns run to completion, every point at which a command can
arrive is a safe point, with one exception: switching is refused while intents
are unresolved (A2). That keeps every pending intent owned by the active work
item, so boot reconciliation (A3) stays single-item.
- **The operator can always switch; the agent may switch on its own in two cases,**
when other work items are ready:
- *The active item is waiting on the operator* (an open gate, a pending question,
or a halt in `AWAIT_INPUT`). An alert must not sit for hours behind an
unanswered prompt. The item it left stays waiting; switching answers nothing
on its behalf.
- *The active item has closed* (completed or abandoned). Ready work should not
sit behind an idle agent until the operator notices.
Either way, it posts a notice naming both items and why it chose that one. The
agent never switches away from an item that is working.
- **A grace period comes before switching away from a waiting item.** After the
active item posts a plan or a question, the agent waits a configurable period
(default 10 minutes) before switching. An operator who is watching answers in
that time and the item carries on without a switch. Otherwise their approval
could sit behind a long milestone on the other item: once parked, an item only
becomes active again when the active one waits or closes. Closing an item needs
no grace period.
- **A parked item is ready when the agent can act on it without the operator:** it
has not started yet (a duty finding, a ticket), or intake arrived while it was
parked that answers what it was waiting for (an approval, an answer, a review
comment). An item still waiting on the operator, with nothing new, is not ready.
- **The model chooses which ready item to switch to.** This is one of the few
judgments it is trusted with outright, because choosing grants nothing: every
item still meets its own gates, and a poor choice costs time, not safety.
- *It is an agent-level call,* belonging to no work item (A10), made by the
`executor` role at the current config (A11 pins roles per item, and this call
has none).
- *It sees summaries rendered by code from the state store,* one per ready item:
source, age, severity, current state, title. It never sees another item's
transcript, so work items stay isolated (A10).
- *It falls back to the oldest ready item* if the call fails or its answer does
not name one of them. Switching never waits on a model.
- *The switch is one state change* (A9 `work_item.switch`), naming both items,
the reason, and whether the model, the fallback, or the operator chose.
- **Parking gives up authority.** Parking a work item unmounts any elevated MCP
instances (F3) its approved plan mounted. They are remounted when it becomes
active again, if its approval has not expired (C2). An approved plan's
credentials never serve another item.
- **Nothing is lost while parked.** Intake for a parked work item, such as comments
on its Gitea issue or PR, is read from a durable per-thread cursor when it becomes
active again. It is not dropped, and it is not replayed from the beginning.
- **The state schema must not assume one item.** Work items live in their own table,
and "active" is a pointer to one of them. Allowing more than one active item later
(Part H) is then a control-loop change, not a migration of every stored item.
---
## PART A — THE AGENT LIBRARY (`nisse-core`)
The shared spine. Every component here is consumed by both agents. This is the part
@@ -112,8 +242,9 @@ canonical log (A10) is the sole source of truth.
process restart. This is what makes the whole system safe to redeploy and reboot; it
blocks everything else.
Must persist: active project/task, per-project state-machine position, approvals
granted, pending questions, session continuity, (dev) git clone locations.
Must persist: the active work item and each work item's state-machine position
(0.2), approvals granted, pending questions, session continuity, and scope state such
as (dev) git clone locations.
Disposable (never on the state volume): agent code, build artifacts/test containers,
shipped-out logs.
@@ -152,10 +283,18 @@ Every startup (clean or post-crash), before accepting operator commands:
(tree diff for edits, Gitea query for pushes), or on any branch/log mismatch. The
judgment is an observable check, never model self-assessment — the same reasoning
as A12.
6. Any other interruption → move to **`AWAIT_INPUT`** and post to transport
6. **Interrupted mid-sequence (C2) — resume only on evidence.** An approved step
sequence may interrupt itself: rebooting the host the agent runs on is a
legitimate step. On restart, **continue** the sequence only when the intent log
names exactly which step was in flight and that step's expected outcome checks
out against the live system, and the approval has not expired. Anything else
(no recorded outcome check, a failed check, or more than one step
possibly in flight) gates to `AWAIT_INPUT`. The step counts as completed only
by that check, never by assuming it ran.
7. Any other interruption → move to **`AWAIT_INPUT`** and post to transport
(e.g. "restarted mid-milestone-2, branch at commit abc123, continue or re-plan?").
Never blind-resume.
7. Otherwise resume the last committed state.
8. Otherwise resume the last committed state.
### A4. Control Loop, State Machine, and Gating Primitives
@@ -194,9 +333,23 @@ Requirements:
for copy-paste. Any other reply/comment is `reject <feedback>`. Reactions/emoji are
never an approval path. Approvals are consumed at first sight; later edits to an
approval comment are ignored and logged.
- **Approvals bind to a work item, too (0.2).** A bare `approve` binds to the
*active* work item's latest revision, under the unambiguity rule above. An
approval arriving on a *parked* item's external record (a Gitea or Vikunja
comment) is delivered when that item next becomes active. So is a Matrix reply to
a parked item's plan or prompt: the reply relation binds it to that item. Neither
is ever applied to the active item.
- **Autonomy inside an approved unit of work:** all tool/command use auto-approved
(Claude Agent SDK `bypassPermissions` or an allow-all `canUseTool`). Escalation to
the human happens *only* via `ask_human`.
the human happens *only* via `ask_human`. For the Sysadmin Agent, "approved"
means the step list of C2. Only calls that match an approved step are
auto-approved.
- **Kill switch.** The operator command `stop` works in every state, for every
agent. It ends the current turn or delegation at the next tool boundary, unmounts
elevated MCP instances (F3), and moves to `AWAIT_INPUT`. The unfinished delegation
or step sequence is then handled like any other halt: continue, re-plan, or
abandon, on the operator's word. Unattended work needs a way to end it that does
not depend on the model cooperating.
- The loop must survive restart mid-work (relies on A2/A3).
### A5. LLM Access *(tiered)*
@@ -236,8 +389,10 @@ Requirements:
- **No vendor credentials on the agent VM.** LiteLLM holds every key, sourced from
Bitwarden machine accounts (A7); the agent holds one LiteLLM token. Adding a vendor
does not touch the agent VM, and a compromised agent leaks no provider keys.
- Config from env/`.env`, **nothing hardcoded**: base URL, auth token, and the role
mapping (A11). Changing any of it is an env change + restart, nothing more.
- Config from the agent's config file, **nothing hardcoded**: base URL, auth token,
and the role mapping (A11). Secrets in it are references to environment
variables, never values. Changing any of it is a config edit + restart,
nothing more.
- **Budget caps and spend alerts live on the LiteLLM virtual key**, not in agent code.
### A6. Browser Tool
@@ -261,9 +416,13 @@ Requirements:
Requirements:
- Client for **Bitwarden/Vaultwarden** via **machine account / Secrets Manager** (or
`bw` CLI + API key). Not the personal vault.
- Reads only from a **dedicated, minimally-scoped machine account** per agent.
- A single **bootstrap token** (the machine-account credential) is the one secret
that lives on the box: tight file perms, never in image or repo.
- Reads only from a **dedicated, minimally-scoped machine account** per agent, **plus
one per MCP server instance** (F3). Each (service, site, privilege) instance holds
its own upstream credential, so it gets its own machine account, and the agent's
account cannot read an instance's credential.
- Each machine account has a single **bootstrap token**. These are the only secrets
that live on the box: tight file perms, never in image or repo. Each token is
available only to its own container.
- (Deferred, not first cut: dynamic/short-lived secrets via Vault/OpenBao. Leave the
interface able to grow toward this; don't build it yet.)
@@ -298,9 +457,12 @@ by replaying the operational trail:
- `ts` — UTC ISO-8601 timestamp
- `agent` — `nisse-dev` | `nisse-ops`
- `event` — dotted type from the taxonomy below
- `task_id` — correlation ID for the enclosing unit of work, minted when the
unit starts (`use <project>`, milestone start, ticket pickup) and threaded
through every event it causes; `null` only outside any unit (e.g. boot).
- `task_id` — the enclosing work item's ID (0.2), minted when the work item
opens (`use <project>`, ticket pickup, an operator question, a duty's finding)
and threaded through every event it causes; `null` only outside any work item
(e.g. boot, a duty run before it yields a finding, or choosing which item to
switch to). A milestone is a phase
within a work item, so starting one does not mint a new ID.
- `cause` — what triggered the mutation: operator command, approval
(`rev-N` + channel), escalation trigger, reconciliation verdict, or crash
recovery.
@@ -311,6 +473,7 @@ by replaying the operational trail:
|---|---|
| `state.transition` | `from`, `to`, `project` |
| `intent.begin` / `intent.commit` / `intent.abort` | `intent_id`, `kind` |
| `work_item.switch` | `from` (parked or closed item, or `null`), `to`, `reason`, `chosen_by` (`model` / `fallback` / `operator`) |
| `gate.open` / `gate.approve` / `gate.reject` | `gate`, `rev`, `channel` |
| `boot.reconcile` | `action`, per-intent verdicts |
| `schema.migrate` | `from_version`, `to_version` |
@@ -338,6 +501,15 @@ The durable record is a **canonical event log** in SQLite, not a provider-shaped
| `knowledge_retrieved` | document text as read, document ID, revision, retrieved-at (A17) |
Rules:
- **The log is scoped to a work item (0.2).** Every event carries its work item's ID,
and a model call for one work item renders only that item's events. Evaluation
capture (A16) may read across items.
- **Some calls belong to no work item,** and their events go to an **agent-level
scope**, with no work item ID: a duty run until it yields a finding, and the
choice of which item to switch to (0.2). Each such call renders only its own
events plus what code gives it. Agent-level events are never rendered into a
work item's call, and a work item's events reach an agent-level call only as a
code-rendered summary.
- **Canonical tool-call IDs are minted locally.** Provider IDs are mapped at the
boundary and never used as a join key. Providers differ in both ID format and in
where tool calls sit structurally (content blocks vs. a separate array on the
@@ -496,7 +668,13 @@ subsystem.
### A15. Local Model Accommodations
- **Constrained decoding for tool-call JSON.**
- **Constrained decoding for tool-call JSON.** Requested with `tool_choice`
(`required`), and only of models whose LiteLLM entry declares
`supports_tool_choice`: enforcement is the model server's, and asking one that
can't honour it can be worse than not asking. Where it isn't available, the
loop enforces the contract itself: an explicit `finish` tool ends the work, a
prose reply is a violation, and a call to a tool not offered counts as a
failure (A12).
- **Reduced tool schema for the executor role:** fewer tools, flatter arguments, no
deeply nested optionals. Maintain **two tool manifests keyed on role**, rendered at
request time like everything else.
@@ -536,33 +714,24 @@ Part D describes the service; this defines the contract.
desynchronise from the project.
**Agent-owned knowledge is always a cache, never authority.** An agent may rely on
it to know *what exists and where to look*; it must confirm a specific fact
against the live system *before acting on it*. Hard requirement for anything in
C2's confirm-first tier; advisory for read-only and reversible actions.
it to know *what exists and where to look*. It must not rely on it for the facts an
action depends on. A prompt cannot enforce that: the model cannot be prevented from
*believing* a fact it has read. The enforcement is elsewhere, and it is structural:
- **Step preconditions (C2) carry the weight.** Every elevated write is an approved
step, and a step that depends on a fact states it as a precondition, checked
against the live system immediately before the step runs. A stale world model
can produce a wrong plan, but it cannot turn that plan into an action on a
system that no longer matches it.
- **Provenance at the human gate.** When a plan is drafted from recalled facts, the
approval message says so: "target recalled from the world model, observed
2026-08-02" versus "confirmed by `zfs list` two minutes ago". The operator is
the authority there; this makes the evidence quality visible.
- **Audit for the rest.** Retrievals are logged (A10), so a wrong fact can be traced
to the actions that followed it.
**Enforcement is structural, because a prompt is not enforcement.** "Treat this as
a cache" is exactly the advisory guarantee A4 and A12 reject, and the model cannot
be prevented from *believing* a fact it has read. The enforceable goal is
narrower: **an unverified belief must not become an irreversible action.**
1. **The store decides what it serves.** Facts carry provenance and an observed-at
timestamp; retrieval marks or withholds anything past a staleness bound for
high-consequence classes.
2. **Verification as a precondition, checked against the log.** For confirm-first
actions, refuse unless a verifying observation exists *in this task's canonical
log*, recorded after the task began, naming the action's target. Inspectable
without asking the model anything — the same move as A12's observable triggers.
Approximate by nature, so build it for the confirm-first tier only.
3. **Provenance at the human gate** — the primary reliance, because C2 already
stops these actions for confirmation. The prompt states evidence quality:
"target recalled from the world model, observed 2026-08-02, not verified this
session" versus "confirmed by `zfs list` two minutes ago". The operator is
already the authority there; this makes the risk legible rather than invisible.
4. **Audit, not prevention, for the cheap tiers.** Log which facts were retrieved
and which actions followed, so a wrong fact is traceable afterwards.
Residual risk, stated plainly: a model can retype a fact from context with no
retrieval event, and no mechanism here catches that. That is why the world model
stays out of the authority path entirely.
**Dropped:** withholding facts past a staleness bound at retrieval, and refusing
confirm-first actions without a verifying observation in the log. Both were
approximations of what step preconditions (C2) now check directly.
**Never an instruction channel, and never an approval channel.**
- **Task instructions never come from the knowledge store.** Work is driven by
@@ -593,15 +762,14 @@ a model rewriting a precise fact loses precision invisibly.
log** — the text, plus document id and revision for provenance — never referenced
by handle alone: A10 forbids a provider handle being the only reference to content,
and these documents are mutable. Snapshot once per task at first use, not per turn.
**A3 consequence:** a task resumed across a restart carries the old snapshot, so
boot reconciliation re-fetches snapshotted knowledge and flags divergence rather
than silently continuing on a stale copy.
A task resumed across a restart carries the old snapshot; step preconditions, not
a re-fetch, are what stop it acting on a stale copy.
**Retrieval is structural first, discretionary second** — the A12/A14 pattern.
- Procedures are **bound to C2 action classes**: the policy taxonomy and the
procedure index are the same list. Binding is deterministic and auditable
("which procedure governed this action?" has an exact answer), and costs nothing
extra because the taxonomy must exist anyway.
- Procedures are **bound to entries in the C2 policy file** (a tool, or a protected
target), and fetched when a plan that uses that entry is drafted. Binding is
deterministic and auditable ("which procedure governed this action?" has an
exact answer), and costs little because the policy file must exist anyway.
- A model-invoked **search** exists *in addition* for discovery, never as a
replacement for the bound retrieval — a procedure the model may skip does not
constrain it.
@@ -609,10 +777,10 @@ than silently continuing on a stale copy.
**Outage posture is asymmetric.** Unlike B2's poller, "never fails a task" does not
generalise here.
- **Write failure never blocks:** retry, or record locally and reconcile.
- **Read failure for a procedure bound to a confirm-first action gates.**
Proceeding is not a delayed reaction; it is acting unconstrained on precisely the
operation someone wrote a procedure for.
- Read failure for read-only or reversible actions proceeds, logging the gap.
- **Read failure for a bound procedure gates the plan.** Planning without it is not
a delayed reaction; it is acting unconstrained on precisely the operation someone
wrote a procedure for.
- Read failure for anything else proceeds, logging the gap.
**Accepted single point of failure.** Once the agent is the sole author and Outline
the only home, losing Outline loses the world model. No local mirror: infrastructure
@@ -620,7 +788,8 @@ facts are re-derivable by observation, which is the same premise the cache rule
rests on.
**Deferred:** the operator-writable procedures collection. The interface is defined
here; the content and the C2 binding list arrive with the policy taxonomy (Part H).
here; the content arrives when there are procedures worth writing, and binds to the
C2 policy file.
**Concrete mapping onto Outline.** Observed in the running instance: a **collection**
carries per-user permissions with three tiers — visible-to (read), read-write, and
@@ -658,9 +827,9 @@ manager. Permissions are collection-scoped, not per-document.
Outline can expose search, read, create and edit over MCP (remote, Streamable HTTP,
enabled workspace-wide). **Not adopted for the retrieval path**, because A17's
enforcement depends on this library sitting *between* the model and the store: staleness
marking, snapshotting into the canonical log, binding procedures to action classes, and
gating a failed confirm-first read all happen to a retrieval before the model sees it.
contract depends on this library sitting *between* the model and the store:
snapshotting into the canonical log, binding procedures to C2 policy entries, and
gating a failed bound read all happen to a retrieval before the model sees it.
MCP's purpose is the opposite — direct model access — and a procedure the model may
decline to fetch does not constrain it (A12).
@@ -672,12 +841,56 @@ membership, and that the workspace-level toggle does not force broader exposure
intended.
### A18. Prompt Injection
**Assume any model that reads untrusted content may be compromised.** There is no
robust way to tell a model "this is data, not instruction". Instructions and data
reach it as one token stream, and every marker meant to separate them is text that
the data can imitate or argue against. Role separation, delimiters and detection
classifiers lower the success rate of injection. None of them bring it to zero, and
an attacker gets to try repeatedly. So the design does not depend on the model
telling them apart. It limits what a compromised model can do.
Untrusted content, for these agents: alert payloads, log lines, ticket and comment
text, web pages the browser reads, repository contents, and downloaded packages.
Requirements:
- **Gates hold against a compromised model.** Nothing a model says can approve a
plan (A4), widen what a plan authorizes (C2), or reach a credential that was not
mounted (F3). Every one of those rules is enforced in code, outside the model.
- **Approval messages are rendered by code from what will actually run** (C2),
never from a model's description of it. Otherwise a compromised model can write
"restart Gitea" above a step that destroys a dataset.
- **Instruction channels are operator-scoped.** Only allowlisted identities can
drive work: operator commands (A1), PR review comments (B2), action-triggering
tickets (C4). Content from anyone else is data.
- **Parse intake into structured fields in code** wherever the source allows it,
e.g. alert name, severity, host, from Alertmanager's JSON. Less free text then
reaches the model at all, and what reaches it is labelled by where it came from.
- **Some injection is accepted because it can only reorder work.** An alert title
can tell the switch decision (0.2) to pick it first. The worst case is ordering,
and the switch notice makes it visible.
- **Check each agent against the lethal trifecta:** access to private data, exposure
to untrusted content, and a channel to send data out. Damage needs all three.
- `nisse-dev` reads untrusted content and holds little private data: its Gitea
token and the repos it works on. Its only ways out are Gitea and package
registries (B3). The registries are an accepted, narrow residual channel.
Routing them through a local mirror would close it, if that ever becomes worth
doing.
- `nisse-ops` has all three: it reads alerts and logs, holds infrastructure
credentials, and has broad egress. It relies on the gates above, not on
removing a leg.
---
## PART B — THE DEV AGENT / `nisse-dev` (build first)
A thin app over `nisse-core`. Plans and implements software projects; produces PRs.
In 0.2's terms, a **project** below is a dev work item on a repo scope. `use
<project>` opens or resumes a work item on a repo, and `project complete` closes the
work item. The repo, its clone, and any other work items on it continue.
### B1. Workflow (state machine)
```
@@ -700,10 +913,12 @@ PR_OPEN
### B2. Requirements
- **One active project at a time.** Operator selects it.
- **One active project at a time** (park-and-switch, 0.2). The operator selects it,
or the agent switches on its own under 0.2's rules; other projects stay parked at
their last position and resume where they left off.
- **Operator commands** (over transport): `use <project>`, `approve`,
`reject <feedback>`, `start milestone <n>`, `status`, plus free text during
planning (forwarded to the agent as answers).
`reject <feedback>`, `start milestone <n>`, `status`, `stop` (A4), plus free
text during planning (forwarded to the agent as answers).
- **Plan gating** per A4: architecture plan and each milestone plan require sign-off
before the next phase. Enforced structurally. Plans and approvals are externalized
to Gitea issue comments (Part H, decided); Matrix and Gitea are both approval
@@ -746,13 +961,16 @@ PR_OPEN
### B3. Deployment & Isolation
- **Dedicated VM** (Proxmox), not a bare container — the agent builds and runs Docker
containers, so it needs its own real Docker daemon. (Docker-in-a-container via
socket-mount ≈ host root; via dind = nested-storage pain. The VM boundary is the
correct trust boundary for unattended arbitrary command execution.)
- **Sealed to start:** egress allow-list (LiteLLM, Gitea, package registries), **no
ingress**. Egress denial must include **RFC1918 except the specific LiteLLM/Gitea
IPs** — sealed from the internet must also mean no reach to NAS/Proxmox/UDM.
- **Dedicated VM** (Proxmox), as for every agent (F1). The Dev Agent has an
additional reason: it builds and runs Docker containers, so it needs its own real
Docker daemon. (Docker-in-a-container via socket-mount ≈ host root; via dind =
nested-storage pain. The VM boundary is the correct trust boundary for unattended
arbitrary command execution.)
- **Sealed to start:** `nisse-dev`'s egress allowlist (F2) permits LiteLLM, Gitea,
and package registries, and there is **no ingress**. Egress denial must include
**RFC1918 except the specific LiteLLM/Gitea IPs**, and also the tailnet's
**`100.64.0.0/10`**, which is not RFC1918. Sealed from the internet must also mean
no reach to NAS/Proxmox/UDM.
- **The LAN-resident inference host is not a second exception.** It sits behind
LiteLLM (A5), so the allowlist keeps exactly one LLM destination regardless of how
many models or vendors are in play.
@@ -777,7 +995,7 @@ tickets; gated on consequences. Built after the Dev Agent works.
| Axis | Dev Agent | Sysadmin Agent |
|---|---|---|
| Rollback | revert snapshot / discard branch | none for `zfs destroy` or a firewall lockout |
| Trust model | allow-all in sandbox | tiered **allow / deny / confirm** policy |
| Trust model | allow-all in sandbox | **approved step sequences**, checked on every call |
| Work shape | long milestones → a PR | short reactive tasks + scheduled checks → live changes |
| Driver | Gitea projects | **Vikunja tickets** |
| Placement | sealed, own VLAN, no Tailscale | Tailscale, credentials, broad reach |
@@ -787,27 +1005,100 @@ deployments over the same library.
### C2. Permission Policy Engine *(the defining addition)*
Tiered policy, evaluated on every action:
- **Read-only telemetry** (metrics, logs, status) — free.
- **Reversible service actions** (restart a container) — allowed.
- **High-consequence actions** (storage, networking, auth) — **confirm-first** over
transport before executing.
**Every write requires an approved plan first.** Write credentials exist only in the
elevated MCP server instances (F3), and those are mounted only after plan approval.
Read-only telemetry (metrics, logs, status) goes through the read-only instances,
which are mounted by default, so it needs no plan.
This is the sysadmin analog of the dev agent's plan gate: the human boundary sits at
irreversible / high-blast-radius actions.
This is the sysadmin analog of the dev agent's plan gate. The dev agent has a second
gate after the work (the PR); the sysadmin agent does not, because a live change has
no review step before it takes effect. So its plan gate has to be exact.
Special cases fold into the tiers:
- **Browser actions against admin UIs default to confirm-first.** Changes to
**network gear (UniFi) are the most conservatively gated action in the system** — a
bad remote UI click can sever the path used to fix it.
**An ops plan is a list of literal steps, and approval covers exactly those steps.**
A write plan is submitted through `submit_plan` (A4) as structured data, not prose.
Each step names:
- the **tool and target**, e.g. the elevated Proxmox instance for `garrett-hall`,
node and VM ID;
- the **arguments**, in full. A config change carries the exact diff;
- an optional **precondition**, checked against the live system just before the
step runs;
- an optional **expected outcome**, checked just after, e.g. host reachable again,
config hash matches;
- a **retry allowance**, default one.
One approval can cover a sequence with several high-consequence steps: "reboot the
host, change the config, reboot again" is one plan and one `approve`. The plan may
also carry prose explaining it, which is context for the operator, never scope.
**The approval message is rendered by code, from the steps.** What the operator reads
is what will run; a model's description of a step is never the thing approved (see
A18). Steps that are irreversible, that touch a protected target, or that will take
down the agent's own host are flagged at the top of the message.
**Execution is checked against the approved steps.** Every tool call passes through a
matcher before it reaches an MCP instance:
- **A read-only call is always allowed.** The agent can look around between steps.
- **An elevated call runs only if it exactly matches the next unfinished approved
step.** Arguments are normalized against the tool's JSON Schema before comparison
(types, key order), so a model's formatting cannot cause a false mismatch.
- **Anything else halts:** a call not in the plan, different arguments, steps out of
order, or a step run again after it completed. The agent moves to `AWAIT_INPUT`,
and the operator decides whether to continue, re-plan, or abandon.
- **A failed step may be retried** up to its allowance. A failed precondition or
expected-outcome check halts, without retry: the world does not match the plan,
and improvising around that is exactly what the gate exists to prevent.
- **A step that may have interrupted the agent** (a reboot of its own host) resumes
under A3's sequence rule.
**Approvals expire.** An approved plan that has not started within the expiry window
must be approved again. An approval is a judgment about the world as it was when the
operator read the plan. The window is policy data (Part H).
**Classification is small, conservative, and data.** The policy is a declared,
versioned file, like the egress allowlist (F2). It holds:
- **Instance privilege.** Read-only or elevated, per MCP instance (F3). This alone
decides whether a call needs an approved step.
- **Protected targets.** The NAS, the gateways, the hosts the agents run on, and
anything else whose loss is expensive. Protection changes how a step is
presented, not whether it is allowed.
- **Each elevated tool's target argument** (`vmid`, `node`, a device MAC), so the
matcher and the renderer can tell which target a call touches. An elevated tool
with no mapping is treated as touching a protected target.
Defaults fail safe. Anything unclassified is treated as elevated. MCP tool annotations
(`readOnlyHint`, `destructiveHint`) are declared by the server itself, so they may
make a tool stricter, never looser. A model may propose entries for a new server's
tools offline, as a PR the operator reviews. Nothing classifies at run time.
**Deferred, with the reason:**
- **Bounded matching** for reversible tools: allowing any call within an approved
target and action, not only the exact arguments. Added one tool at a time, where
exact matching causes real friction. It needs a per-tool reversibility
classification, which is the up-front cost exact matching avoids.
- **Standing approvals:** a plan template with a trigger, a rate limit and an
expiry, e.g. "restart Gitea on sustained 502, at most twice an hour, notify
after". The same matcher, run without a fresh `approve`. Added when a routine fix
has been approved by hand often enough to show its shape.
Special cases:
- **Browser actions against admin UIs cannot be matched as steps,** since a click is
not a stable literal. Each one is confirm-first over transport, even within an
approved plan. Changes to **network gear (UniFi) are the most conservatively
gated action in the system**: a bad remote UI click can sever the path used to fix
it.
- **Action-triggering tickets are operator-scoped** (C4). Auto-generated tickets
(e.g. from a monitoring alert) route through confirm-first, never drive unattended.
(e.g. from a monitoring alert) can only lead to a plan, which the operator
approves like any other. They never drive unattended.
### C3. Interfaces
- **MCP servers (F3):** Proxmox and UniFi, at both sites, through per-site,
per-privilege server instances on the agent's own Docker host. This is the primary
path to that infrastructure.
- **Browser (A6):** into admin UIs. **Prefer real APIs where they exist** (Proxmox,
TrueNAS good; UniFi partial); use the browser only for genuinely UI-only gaps —
it's the least stable interface.
TrueNAS good; UniFi now reachable over MCP); use the browser only for genuinely
UI-only gaps. It's the least stable interface. For network gear, it is a fallback
for API gaps, not the primary path.
- **SSO (A8):** logs into admin UIs as `svc-nisse-ops`. This account is a high-value
credential; its blast radius equals its Authelia scope, so scope it minimally.
- **Secrets (A7):** UI/SSO logins and service API tokens from Bitwarden.
@@ -818,8 +1109,12 @@ Special cases fold into the tiers:
### C4. Ticket-Driven Control (Vikunja)
The sysadmin analog of "dev agent driven by Gitea."
- **Vikunja ticket = task of record** (durable, has status/queue/history). **Matrix =
live conversation about it** (questions, confirm-first prompts, progress).
- **Vikunja ticket = task of record** (durable, has status/queue/history): the
external record of a work item (0.2). **Matrix = live conversation about it**
(questions, halts, progress).
- **Scheduled checks and alert intake are duties (0.2).** Tickets they generate are
duty findings, deduplicated against open tickets, and like any auto-generated
ticket they can only lead to a plan the operator approves (C2).
- **Flow:** ticket filed → agent picks up → discuss/confirm over Matrix → update
status as it progresses → close with result.
- **Mechanics:** Vikunja REST API + webhooks (same two-direction, JSON-only pattern
@@ -827,16 +1122,19 @@ The sysadmin analog of "dev agent driven by Gitea."
installed version's webhooks fire on comment creation; if not, poll comments as
B2 polls Gitea.
- **Approvals:** Vikunja is an `ApprovalSource` (A4) for plan gates on its own
tickets, via `approve rev-N` comments. Live **confirm-first** prompts (C2) stay
Matrix-primary — polling latency makes a ticket comment a poor interactive
channel — with the outcome written back to the ticket as the record.
tickets, via `approve rev-N` comments. Live prompts (C2's halts and browser
confirmations) stay Matrix-primary — polling latency makes a ticket comment a poor
interactive channel — with the outcome written back to the ticket as the record.
- **Trust:** a driving ticket is an instruction channel into an infrastructure-
changing agent → action-triggering ticket creation is operator-scoped (C2).
### C5. Deployment & Isolation
- Longer-lived, **stateful VM** (not disposable).
- Longer-lived, **stateful VM** (not disposable), dedicated to this agent (F1).
- Has **Tailscale** + scoped credentials to reach infrastructure across both sites.
- **Its MCP servers (F3) run on its own Docker host.** Its egress allowlist (F2)
therefore includes the Proxmox and UniFi endpoints at both sites, even though the
agent process never opens those connections itself.
- Updated **in place** via the container pipeline, **restart gated on an idle flag**.
---
@@ -899,7 +1197,89 @@ each agent is built — not standing up the service itself.
---
## PART F — DEPLOYMENT MODEL (both agents)
## PART F — DEPLOYMENT MODEL (all agents)
These rules apply to every agent, current and future. Parts B3 and C5 record how each
existing agent instantiates them.
### F1. Agent hosts: every agent gets its own VM
- **Every agent gets its own VM** (Proxmox). No agent runs as a container on a shared
host, and no two agents share a VM.
- **Why, in general:**
- *Isolation boundary between agents.* The VM is the trust boundary (Part H). A
compromised or misbehaving agent must not reach another agent's state,
credentials, or processes.
- *Per-agent credential attribution.* Everything an agent holds lives on its own
host and is issued to its own identity, so every upstream action traces back to
exactly one agent.
- **Agent-specific reasons add to this; they are not its foundation.** The Dev Agent
also needs a real Docker daemon, because it builds and runs containers (B3). An
agent that builds no code still gets a VM, for the two reasons above.
### F2. Egress policy: per agent instance
- **Every agent instance has its own default-deny allowlist.** Requirements differ by
agent type. No agent inherits another agent's policy.
- **The allowlist is a declared artifact,** versioned alongside the agent's
definition. Each entry is justified by the component that needs it.
- **The declaration point is fixed. The enforcement point is not.** Policy may be
enforced at the VM, at the Docker network, at the UniFi firewall, or at several of
these at once. Layering is deliberate: enforcement at the network layer survives
compromise of the agent's own host.
- **The allowlist is the union** of what the agent reaches directly and what its MCP
servers (F3) reach on its behalf. `nisse-ops` must permit Proxmox and UniFi
endpoints at both sites, even though the agent never opens those connections
itself.
- **The tailnet is `100.64.0.0/10`** (CGNAT shared address space), not RFC1918. A
rule written around RFC1918 denial does not cover it. Deny it explicitly.
### F3. MCP servers
**One instance per agent VM.** MCP servers run as containers on the owning agent's
Docker host. Images are shared through the OCI registry. Processes are not shared
between agents.
- *A shared process means a shared credential.* The upstream API's own audit log
then cannot attribute a call to a specific agent. Separate instances with separate
tokens give attribution at the API layer, instead of depending on our own
transcripts. This matters most for a read-only auditing agent, such as a future
`nisse-security`.
- *An agent should not host the tooling used to inspect it.*
- *Duplication is nearly free.* These servers wrap remote HTTP APIs and hold no
state. (Stateful servers are the exception that could justify relocation. That
item is parked in Part H.)
**One instance per (service, site, privilege level).**
- **Site belongs in the server's identity,** using the site identifier (Part 0), not
in a `site` tool argument. A wrong-house action should require a tool that isn't
mounted, not just a wrong parameter value.
- **Read-only and elevated access are separate instances** with separate upstream
credentials, each from its own machine account (A7). The read-only instance is
mounted by default. The elevated instance is mounted only after plan approval (C2).
This puts the capability gate at the credential layer, instead of a flag inside a
third-party server.
- **Per-task tool profiles** mount only the servers that a given task's scope
requires.
#### Rejected alternative: routed meta-tools / lazy tool registration
Several community Proxmox and UniFi MCP servers use search-then-dispatch over a large
generated tool catalog. This is **rejected in favor of per-task tool profiles**:
- *Two extra round-trips per call,* which is worse with a local model driving the
control loop.
- *The model can no longer discover capabilities it wasn't already looking for.*
That is backwards for exploratory diagnostic work.
- *Arguments arrive as an opaque blob,* instead of being constrained by per-tool JSON
Schema.
- *A dynamic tool list defeats prompt prefix caching,* and caching is much of what
makes a large static list affordable.
Acceptable only when consuming a third-party server whose surface we can't trim.
### F4. Containers, images, and restarts
- **Containers + persistent volumes.** Stateless image; named volumes carry state
across redeploys (and allow optional persistence between deployments).
@@ -907,6 +1287,9 @@ each agent is built — not standing up the service itself.
- **Gate the restart, not the pull:** an idle/"safe-to-restart" flag; Watchtower must
never restart mid-work. *Unplanned* interruptions (reboot/crash) are handled by the
durability layers (A2/A3), not by restart timing.
- **Carve-out: third-party MCP server images** (F3) are excluded from Watchtower.
They hold root-equivalent credentials to Proxmox and UniFi, so they are
digest-pinned and updated deliberately after review.
- **Survive reboots:** VM + Docker service set to start on boot; `restart:
unless-stopped`; ZFS snapshots of the state dataset (scheduled + on each approval).
- **Terraform + cloud-init:** deferred. When adopted, `bpg/proxmox` + cloud-init,
@@ -936,7 +1319,7 @@ each agent is built — not standing up the service itself.
10. **Extract `nisse-core`**; build the **Sysadmin Agent** (`nisse-ops`): policy
engine (C2), SSO + scoped browser (A8/A6), secrets (A7), Vikunja control (C4),
knowledge store (A17/D) — world model first, operator procedures when the
C2 taxonomy they bind to exists — scheduled checks, alert intake.
C2 policy file they bind to exists — scheduled checks, alert intake.
- With the monorepo layout (Part H, decided), extraction is a **package split**
within one tree: move code behind a package boundary, no release machinery
required.
@@ -976,10 +1359,24 @@ The rest resolve when reached, not now.
are still moving. One repo can still build both agent images via path-filtered
CI. Splitting later stays easy if a real need appears; merging later would not
have been.
- **Sysadmin browser access to network gear:** standing vs. per-task hand-off. The
lock-yourself-out risk argues for per-task.
- **Policy action taxonomy (C2):** the concrete list of which operations are
allow vs. confirm. Best decided with the agent half-built in front of you.
- **Sysadmin fallback browser access to network gear:** standing vs. per-task
hand-off. With UniFi reachable over MCP (F3), the browser is no longer the primary
path to network gear. It is a fallback for operations the UniFi API does not
expose. The open question is whether that *fallback* access is standing or handed
off per task. The lock-yourself-out risk still argues for per-task.
- **Ops gating model (C2) — decided: approved step sequences, exact matching.** A
per-tool allow/confirm taxonomy was the original plan. It would have to be
written in advance for a tool catalog that will grow large, and every mistake in
it is either friction or a hole. Approving literal step sequences makes most of
it unnecessary: every elevated write has to be an approved step anyway, and
classification is left to decide only how an approval is *presented* (protected
targets, irreversible steps). What remains open is data, best filled in with the
agent half-built in front of you: the protected-target list, each elevated
tool's target argument, and the approval expiry window.
- **Parked: bounded matching and standing approvals (C2).** Both loosen exact
matching and reuse the same matcher. Add bounded matching one tool at a time,
where exact matching causes real friction. Add standing approvals when a routine
fix has been approved by hand often enough to show its shape.
- **Approved-plan externalization — decided: Gitea-backed from the start.** Plans are
posted as issue comments on the project repo; the approval comment (author,
timestamp, revision tag) is the durable off-box record, authenticated by Gitea's
@@ -987,6 +1384,36 @@ The rest resolve when reached, not now.
so boot-time reconciliation (A3) can re-derive gate state from Gitea even after
state-volume loss — external truth wins (A2). Approval mechanics and revision
binding are specified in A4.
- **Parked: relocating stateful MCP servers off the owning agent's VM.** The trigger
is **statefulness** (a browser pool, a vector index, a warm cache), not sharing:
duplicating a stateful server duplicates its state. Relocating has a cost. A shared
server can't hold per-agent tokens, so per-agent identity has to move into the
request, through caller auth plus per-caller upstream credentials. That loses the
attribution property that makes instance-per-agent-VM (F3) clean.
Instance-per-agent-VM is the starting position. Migrate one server at a time, when
a specific server justifies it.
- **Parked: more than one active work item (0.2).** Park-and-switch is the starting
position. The pressure to relax it will come first from the Sysadmin Agent: an
alert arriving mid-diagnosis, or a duty's scheduled run coming due while a work
item is active, has to wait for a safe point. The agent's own switch (0.2) takes
the edge off: an alert waits for an item that is working, not for an operator
who is away. Allowing concurrent items would cost:
- *Serialization in the state store and control loop.* Transitions for different
items interleave, and the single orchestrator lock becomes per-item.
- *A scheduler.* Something has to decide which item's turn runs, especially with
one local inference host serving all of them.
- *Approval binding.* Two plans or halts open at once make a bare `approve`
ambiguous everywhere, so per-item rooms or strict reply threading become
mandatory rather than convenient.
- *Shared-scope conflicts.* Two active work items on one repo need separate
working trees. Two on one infrastructure scope could make changes that
interfere with each other.
- *Intent ownership.* Boot reconciliation has to sort pending intents by item, and
refusing to switch no longer keeps them single-item.
- **Open: when a work item must have an external record (0.2).** Plan-gated dev work
and ticket-driven ops work already require one. Whether every gated work item
needs one (for instance, an ops question that turns into a change to make)
is best settled once C2's step sequences are in use.
---
@@ -1011,9 +1438,10 @@ nisse-dev/ # app: sealed VM (Part B)
browser → internal test network only
nisse-ops/ # app: separate host, Tailscale (Part C)
policy engine, tiered confirmation, scheduled checks
policy engine (approved step sequences, C2), scheduled checks
Vikunja ticket intake, Outline runbooks/records
browser → admin UIs (API-first, UI fallback), confirm-first gating
MCP → Proxmox/UniFi per site and privilege (F3); runs on its own VM
browser → admin UIs (API-first, UI fallback), confirm-first per action
infra/ # (future) Terraform + cloud-init (Part F)
nisse-dev.tf | nisse-ops.tf | templates/
docs/Tuning-Guide.md
+489
View File
@@ -0,0 +1,489 @@
# Tuning Guide
How to fit a local model onto the inference host, and how to set the agent's
tuning knobs. Part 1 covers the ideas you need to reason about the hardware.
Part 2 covers each knob in `nisse-dev.yaml`. Part 3 gives an order to tune
in.
The knobs live under `tuning:` in the config file (see
`nisse-dev.example.yaml`). They are read at startup, so a change takes effect
on the next restart. Unknown or misspelt keys are startup errors.
---
## Part 1 — How a local model spends VRAM
A model server holding a model in GPU memory spends it on three things:
1. **Weights**, which are fixed for a given model and quantization.
2. **The KV cache**, which grows with context length.
3. **Working memory** for activations and runtime buffers. Budget 1–2 GB of
headroom.
On a 24 GB card, the whole game is trading (1) against (2).
### Weights and quantization
A model's weights take roughly *parameters × bits per weight ÷ 8* bytes.
| | 16-bit (full) | 8-bit | ~4.5-bit (Q4_K_M, AWQ) |
|---|---|---|---|
| 24B | 48 GB | 24 GB | ~14 GB |
| 32B | 64 GB | 32 GB | ~19 GB |
Quantization stores each weight in fewer bits. Common 4-bit formats come out
nearer 4.5–5 bits per weight once scaling factors are counted, so check the
actual file size. The quality cost is small at 5–6 bits, usually acceptable at
4, and it climbs quickly below that. For agent work the cost shows up first as
**worse tool-call discipline**, not as worse prose. The capability matrix
(Part 3) measures exactly that.
**Mixture-of-experts** models such as Qwen3-30B-A3B are different. Memory is
set by *total* parameters (30B), but speed is set by the parameters *active*
for each token (3B). An MoE model of a given size therefore generates much
faster than a dense model of the same size. It's often weaker than a dense
model of the same total size, but not always for tool use.
### The KV cache, and why context costs memory
As the model reads the prompt, every layer stores a key and a value vector for
every token, so that later tokens can attend to them without recomputing.
That store is the KV cache. Its size per token is:
2 (key and value) × layers × KV heads × head size × bytes per value
The numbers come from the model's `config.json` (`num_hidden_layers`,
`num_key_value_heads`, `head_dim`). Some worked examples at 16-bit:
| Model | Layers × KV heads × head size | Per token | 32k tokens |
|---|---|---|---|
| Qwen3-32B | 64 × 8 × 128 | 256 KB | ~8 GB |
| Mistral/Devstral Small 24B | 40 × 8 × 128 | 160 KB | ~5 GB |
| Qwen3-30B-A3B (MoE) | 48 × 4 × 128 | 96 KB | ~3 GB |
(Taken from the published configs; recheck against the model card before
relying on them.)
Two consequences:
- **Context length is a memory decision, not a free setting.** The spec's
estimate (Part H) was about 5 GB left for KV cache beside a 4-bit 32B model.
At 16-bit, that's only about 20k tokens of context for Qwen3-32B.
- **KV-cache quantization** stores keys and values in 8 bits (or 4). At 8
bits it halves the cache and is close to lossless. At 4 bits it is
noticeably worse. Quantizing the cache to 8-bit is usually the first lever
to pull, since it buys twice the context for almost nothing.
The server allocates the cache for the context length you configure. It
doesn't matter how much of it a request uses. So set the context length on the
server deliberately: it is the ceiling for *prompt plus response* on every
request.
### Prompt processing vs generation
A request has two phases, and they run at very different speeds:
- **Prefill** reads the prompt. It is parallel and fast: thousands of tokens
per second on a 4090.
- **Decode** writes the response one token at a time. Roughly 30–40 tokens per
second for a dense 32B at 4-bit, and several times that for a 3B-active MoE.
Treat these numbers as ballpark; measure your own.
An agent loop re-sends the whole transcript on every turn, which makes the
prompt long and the responses short (a tool call is often under 200 tokens).
So prefill matters more than the headline tokens-per-second suggests.
**Prefix caching** is the fix. If the start of a prompt matches the previous
request, the server reuses the KV cache for it and prefills only the new part.
llama.cpp does this per slot, and vLLM does it automatically. For this
workload it is often the single largest speed win. It works here because the
rendered transcript only grows at the end, so consecutive turns share
everything but their newest events.
### Tokens and characters
For English and code, a token is roughly **4 characters**. That's useful for
converting the character-based knobs below into context cost.
---
## Part 2 — The knobs
For each knob: what it controls, what goes wrong at either extreme, and what
to look at when deciding.
### `llm.max_output_tokens` — default 16384
The ceiling on a single response, for every role. The ceiling actually sent
is the smaller of this and the model's own `max_output_tokens` in LiteLLM, so
a local model can be held to less without affecting the frontier roles.
- **Why it exists:** a response that hits the ceiling is truncated
mid-sentence, or worse, mid-tool-call, where it can still parse. The agent
detects truncation and treats it as an error, but a ceiling that's too
tight still costs a failed turn. Planning with a frontier model was observed
using over 4,000 output tokens.
- **Too high, on a local model:** the server reserves room for the whole
response inside the context window. With a 32k context and a 16k ceiling,
only 16k is left for the prompt. Some servers reject any request where
prompt plus ceiling exceeds the context.
- **Too low:** truncated plans and tool calls.
- **Deciding:** executor turns are short, so 2–4k is plenty for a local
executor. Frontier planning wants the large default. Leave this high, and
set a local model's smaller ceiling as its `max_output_tokens` in LiteLLM.
### `tuning.executor.max_turns` — default 40
How many model calls one milestone may make before the executor stops with
`capped` and asks you what to do.
- **Too low:** milestones that were making progress stop short. You'll see
`capped` outcomes on work that was going well.
- **Too high:** a confused model keeps going. Every turn also adds its tool
calls and results to the transcript, so a long run spends context as well
as time.
- **Deciding:** look at how many turns *successful* milestones take, and set
the cap comfortably above the high end, around 1.5–2× the turns of your
longer successful runs. A stronger model usually needs fewer turns, not
more.
- **Measured:** the default was 24 until the harder benchmark tasks arrived.
Haiku, used as the ceiling in calibration, hit 24 on both runs of
`refactor-rename` (editing ten files one at a time) and of
`long-milestone`, one of which had in fact finished the work. It is a guard
against a model trying forever, not a budget, so it was raised to 40.
### `tuning.executor.failure_limit` — default 2
How many consecutive failed `run_command` calls trigger escalation: the
sub-task is handed to a frontier model as a delegation (spec A12). A
successful command resets the count. Failed file reads don't count, since
reading is looking around, not doing the work.
- **Why it's a count and not a judgment:** small models are poorly calibrated
about their own competence. Asked "are you stuck?", they say no until far
too late, or yes constantly. A count of observable failures is a signal
that doesn't depend on the model's opinion of itself.
- **Too low:** normal iteration escalates. Test-first work *expects* a failing
test run before the fix, so a limit of 1 hands off routine work and spends
frontier tokens on it.
- **Too high:** a weak model circles, re-running the same failing command
with small variations, while the transcript fills up.
- **Deciding:** raise it as local models get stronger. Watch two things
together. First, how often delegations fix what the local model couldn't
(escalation is doing its job). Second, how often the local model recovers
on its own after its second failure (the limit may be too low).
### `tuning.expert.consult_budget` — default 3
How many `ask_expert` calls one milestone may make. `ask_expert` is a single
stateless question to a stronger model, with no access to the repository
(spec A14), for things like "is this API deprecated?".
- **Too low:** the executor can't get unstuck on a knowledge question it
can't answer from the code.
- **Too high:** a confused executor asks variations of the same question
instead of doing the work or failing. That's the circling A12 warns about.
- **Deciding:** count consults per milestone, and check whether the answers
changed what the executor did next. If it regularly exhausts the budget
without progress, the budget is fine and the model or the plan is the
problem.
### `tuning.delegation` — hard caps on one delegation
A delegation is a frontier model working a sub-task with full tool access
(spec A13). When a cap is hit, the delegation **fails**, and its branch and
work are discarded, because resuming on a half-edited tree isn't safe.
| Knob | Default | Caps |
|---|---|---|
| `max_turns` | 30 | model calls |
| `max_tokens` | 200000 | input + output tokens, as reported by the provider |
| `wall_clock_s` | 900 | elapsed time |
- **`max_tokens` is a scope limit, not the money limit.** Spending limits live
on the LiteLLM key (spec A5). This cap stops one delegation from growing
without bound. An unbounded one could still exhaust a daily budget and take
every frontier model away until it resets.
- **Too low:** delegations fail just short of finishing and their work is
thrown away. That is the most expensive way to be wrong, because you paid
for the work.
- **Too high:** a stuck delegation burns tokens and time before failing.
- **Deciding:** every delegation records the tokens it used and how long it
ran (the `delegation_completed` event, captured for evaluation per spec
A16). Set each cap above what successful delegations actually use, with
room to spare.
### `tuning.tools.command_timeout_s` — default 120
How long one `run_command` may run before its whole process group is killed.
- **Too low:** slow test suites, dependency installs and Docker builds get
killed and reported as failures. That also counts toward `failure_limit`.
- **Too high:** a command that never exits blocks the turn for the whole
timeout, such as a dev server started in the foreground or a test waiting on
input. `stop` waits for a running command to finish too, so this is also the
worst-case delay before a stop takes effect.
- **Deciding:** set it just above the slowest legitimate command in the
projects the agent works on.
### `tuning.tools.max_output_chars` — default 4000
How much of a tool's output (command output, file contents, directory
listings) the model sees. Longer output keeps its first quarter and last three
quarters, with a note in between saying how much was left out. The end gets
the larger share because test runners and compilers print their summary of
what failed last; the start keeps the exit code.
- **This is a context knob.** Every tool result stays in the transcript and is
re-sent every turn. 4,000 characters is about 1,000 tokens, so 20 turns of
maximal output is about 20k tokens, which is already most of a local
model's context.
- **Too low:** the model doesn't see the part of the output that matters.
- **Too high:** the transcript outgrows the context window sooner. The
context budget copes (see below), but by cutting older output harder.
### `tuning.context` — fitting each request into the window
Before any request is sent, the agent checks it against the model's context
window and makes it fit. A server doesn't always refuse a prompt that's too
long. Ollama silently drops its start, which is where the system prompt and
the plan are, so an unfitted request can fail in a way nobody sees.
The window is each model's `max_input_tokens` in LiteLLM (see *Declaring a
local model's limits* below). A model in use without one stops the agent at
startup. From the window come the output ceiling and a safety margin, and
the rest is for the prompt. When the prompt is too big, three steps apply in
order, stopping at the first that fits:
1. **Older tool results are cut** to about 600 characters, head and tail.
The most recent turns keep their output whole.
2. **The oldest turns are dropped,** tool calls together with their results.
A note at the start tells the model how many events were left out. The
log keeps everything; only the request is shortened.
3. **If the system prompt and the recent turns still don't fit,** the
request is not sent. The milestone stops and asks you.
Tokens are estimated from characters (pessimistically, at 3 per token) and
the estimate is corrected from the prompt sizes the server reports. The
correction only ever makes the estimate larger.
The executor also sends less in the first place: the approved milestone plan
goes in its system prompt, which is never cut, and it sees only what happened
since that plan. The planning conversation and earlier milestones stay in the
log.
| Knob | Default | |
|---|---|---|
| `keep_recent_turns` | 6 | model turns that are never cut or dropped |
| `safety_margin` | 0.1 | fraction of the window held back for estimation error |
- **`keep_recent_turns` too low:** the model loses sight of what it just did
and repeats itself. **Too high:** less room to cut before step 3 gives up;
on a small window, overflows come sooner.
- **`safety_margin` too low:** an underestimate reaches the server, and a
silently cutting server drops the start of the prompt. **Too high:** wasted
context. Lower it only once the correction from reported usage has settled,
which takes a few calls per model.
- **Deciding:** if milestones stop with "cannot fit", look first at
`tools.max_output_chars` and the model's window, not at these.
### Enforced tool calls — `supports_tool_choice`, declared in LiteLLM
The agent's loops depend on every turn being a well-formed call to a real
tool. A model only *writes text*: a "tool call" is text in a format the model
was trained to produce, which the server recognises and turns into a
structured call. By default (`tool_choice: auto`) the server just parses
whatever comes out, so prose, broken JSON and calls to invented tools are all
possible.
With `tool_choice: required`, a server that supports it **constrains
decoding**: at every step it rules out any token that couldn't lead to a
valid call to one of the offered tools, with arguments matching its schema.
The model still chooses the tool and the arguments; it can't produce anything
else. That makes it a property of the **model server** (and sometimes of the
model), not something a prompt can buy.
So the agent asks for it only where it is known to work: models whose LiteLLM
entry declares `supports_tool_choice: true`. Measured on the 4070:
| Alias | Server | `required` on a chat prompt | Declare |
|---|---|---|---|
| `local-fast` (Qwen3-8B-AWQ) | vLLM 0.29 | calls `finish` every time | `true` |
| `local-coder` (Qwen3-14B) | llama.cpp | can't stop; writes until the output ceiling (a known Qwen3-14B bug) | `false` |
| `claude-haiku-4-5` | Anthropic | calls `finish` every time | `true` (LiteLLM already says so) |
Asking a model that can't honour it can be worse than not asking: the
Qwen3-14B turn above costs a full output ceiling of generation, where a plain
prose reply would have cost a few hundred tokens.
Where it isn't enforced, the executor holds the contract itself:
- **`finish` is the only way to end a milestone.** It is refused once if
nothing has been changed yet.
- **A prose reply is a violation.** The model is told once; a second escalates.
- **A call to a tool that wasn't offered counts as a failure** toward
`failure_limit`, like a failed command, so an inventing model escalates in a
couple of turns instead of using up `max_turns`.
The capability matrix checks the declaration: a model declared `true` that
answers a required call in prose fails
`test_supports_tool_choice_is_declared_truthfully`.
### Declaring a local model's limits
LiteLLM knows commercial models' limits. For a local model, declare them in
LiteLLM's config as the model's `model_info`:
```yaml
model_list:
- model_name: local-fast
litellm_params: {model: openai/qwen3-8b-awq, api_base: http://inference-host:8080/v1}
model_info:
max_input_tokens: 32768 # the server's per-request context
max_output_tokens: 4096 # the most one response may use
supports_function_calling: true
supports_tool_choice: true # only once measured; see above
```
`max_input_tokens` is the *whole* window, prompt plus response. The agent
takes the output ceiling out of it. That's how the commercial numbers work
too: Claude's prompt and response together must fit in its 200k.
The number to use is what the **server** allocates per request, not the
model's advertised maximum:
- **The model's ceiling:** `max_position_embeddings` in `config.json`, or
`<arch>.context_length` in a GGUF file. Some models reach further with RoPE
scaling; the model card says whether that needs switching on.
- **llama.cpp / llama-swap:** the `-c` / `--ctx-size` flag, shown as `n_ctx`
in the load log and in `GET /props`. With `--parallel N` each request gets
`n_ctx / N`, so run the agent's models with `--parallel 1`.
- **vLLM:** `--max-model-len`, reported by `GET /v1/models`.
- **Ollama:** 4096 unless `OLLAMA_CONTEXT_LENGTH` or a Modelfile's `num_ctx`
says otherwise. `GET /api/ps` shows the loaded model's `context_length`;
`ollama show` shows only the trained maximum.
Then check the declaration against the server:
uv run python scripts/probe_context.py local-fast
It sends one prompt half again as long as the declared window and reports
whether the server refused it, cut it (and where), or took it all. A cut
below the declaration is the case to fix: prompts between the two would be
silently cut. Measured on the Jetson's `llama3.2`: cut at 4096, matching its
declaration.
---
## Benchmarking models
The capability matrix asks whether a model *can* hold the tool contract. The
benchmark asks whether it *finishes real work*: four canned milestones, each an
approved plan over a small seed project, judged by a hidden check the model
never sees. Success means the check passes, not that the model said it was
done.
| Task | What it exercises |
|---|---|
| `write-function` | write a small function and its tests |
| `fix-bug` | find and fix the bug a failing test exposes, without touching the test |
| `multi-file` | thread a new option through several files |
| `long-output` | find one failure in test output far longer than `max_output_chars` |
It runs the executor exactly as the agent does (the same client wiring,
context budget and `tool_choice` handling, and your config's tuning knobs),
with **delegation and `ask_expert` turned off**. An escalation is a failure of
the model under test, not a frontier model's success.
**Evaluating a new model** is one command, once its LiteLLM entry declares its
limits:
scripts/evaluate_model.sh qwen3-32b-q4 "vLLM 0.29, AWQ, fp8 KV" 5
It runs the context probe, then the capability matrix, then five runs of
every task, and stops at the first step that fails. Or run the steps
yourself:
uv run python scripts/benchmark.py run --models A,B --runs 5 --note "how it's served"
uv run python scripts/benchmark.py report [--models A,B] [--tasks T]
**Results are kept.** Every run is one line in `bench/results/`, committed
with the repo, recording:
- the model's backing name and declared facts;
- your note on how it was served;
- the task's content hash;
- the agent's commit and tuning;
- the outcome, turns, failures, tokens, time and cost.
The report rebuilds the table from every stored run. Benchmarking a new model
means running only that model and reading it next to everything before it.
Runs against an older version of a task are marked `(old)`. Runs of one model
served differently (different note) are kept apart.
**Reading it:**
- **Compare the intervals, not the rates.** 3 of 5 (23–88%) and 5 of 5
(57–100%) overlap: five runs don't separate them. Add runs until the
intervals stop overlapping, or accept that the models are close.
- **Look at why runs failed**, not just how many (`outcome_note` and the
counts in each result line). An early shakedown on `local-fast` failed
mostly by escalation: the model was fixing its own mistakes and hit
`failure_limit: 2` mid-recovery. That says as much about the knob as about
the model. Tuning a knob and re-running is a fair comparison, because the
tuning is recorded with every run.
- **Cost** is per run, straight from LiteLLM, so a frontier model as executor
can be compared on success rate *and* dollars.
---
## Part 3 — An order to tune in
Upstream choices change what's right downstream, so tune in this order and
change one thing at a time.
1. **Model and quantization.** Start with whether a candidate holds the
tool-call contract at all, and whether its server enforces `tool_choice`:
uv run pytest -m integration tests/integration/test_model_capability.py \
--capability-models=alias-a,alias-b
A model that falls back to prose instead of calling tools is disqualified,
however good its benchmarks look. Before comparing aliases, check what
each one actually points at in LiteLLM (`/model/info`, the
`litellm_params.model` field). Two aliases believed to be the same model
once weren't, and that cost a lot of debugging.
**Make sure LiteLLM has no fallback for a model you are measuring.** A
fallback answers a failed request with another model, under that model's
name, in a response that otherwise looks normal. One made a local model
that couldn't call tools at all look like it "held the tool contract 20 of
20": every answer was Claude's. The capability matrix and the probe now
check which model answered and fail on a mismatch, but the fallback also
quietly turns local work into frontier spending in production.
2. **Context length and KV-cache precision** on the server. Take the largest
context that fits beside the weights with 1–2 GB of headroom. Try an 8-bit
KV cache before giving up context.
3. **Declare the window** in LiteLLM and check it with the probe, then set
the model's `max_output_tokens` there so the prompt keeps most of the
window.
4. **The benchmark.** Run it and look at the evidence: success rate, turns,
escalations, and time taken.
5. **The loop knobs**: `failure_limit`, `max_turns`, `consult_budget`, the
delegation caps, then the tool limits, one at a time.
**Expect noise.** Local model behaviour varies a lot from run to run. One
early comparison measured 12 of 20 runs holding the tool contract against 20
of 20, a real difference (p ≈ 0.003). Three runs each could easily have
pointed the other way. Before changing a knob because of a result, repeat the
run enough times that the result would survive a coin flip.
---
## Coming next
Later work that will add knobs, each with its own section here:
- **Compaction.** A model-written summary replacing old turns, instead of
dropping them. The log already has an event for it; it needs a summarizing
model and a policy for what is safe to summarize.
- **Benchmarking the other roles:** planner and delegate, with delegation
turned back on for comparisons across frontier tiers.
nisse-dev.example.yaml
+60
View File
@@ -0,0 +1,60 @@
# Example configuration for nisse-dev. Copy to nisse-dev.yaml (or point
# NISSE_CONFIG at your copy) and edit.
#
# - Structure lives here. Secrets do not: write them as ${VARIABLE} references,
# resolved from the environment (or .env, see .env.example) at startup.
# - Keep this file outside the workspace clone. Startup refuses otherwise,
# because the agent commits and pushes everything in the clone.
# - Unknown keys are errors, so a misspelt knob fails at startup instead of
# silently doing nothing.
state_db: ~/nisse-demo-workspace/state.db
workspace: ~/nisse-demo-workspace/project # the clone
project: demo
llm: # A5: LiteLLM is the only LLM destination
base_url: http://litellm.example:4000
token: ${NISSE_LLM_TOKEN}
# Ceiling on each response. The model's own max_output_tokens in LiteLLM
# caps it further, which is where to hold a local model to less.
max_output_tokens: 16384
roles: # A11: role -> LiteLLM model alias
executor: local-fast # the local tier; the benchmark picks the model
planner: claude-opus-5-5 # high-end judgement: plans a weaker executor follows
delegate: claude-opus-5-5 # takes over what the executor got stuck on
consultant: claude-haiku-4-5 # one question at a time, no tools: cheap is enough
matrix: # A1
homeserver: https://matrix.example
user_id: "@nisse-dev:example"
token: ${NISSE_MATRIX_TOKEN}
room_id: "!roomid:example"
operators: ["@you:example"]
gitea: # B2
url: https://gitea.example
token: ${NISSE_GITEA_TOKEN}
repo: owner/demo-repo
operators: [your-gitea-login]
base_branch: main
# What each knob does, and how to choose a value: docs/Tuning-Guide.md.
# These are the defaults; the section can be left out entirely. The file is
# read at startup, so a change takes effect on the next restart.
tuning:
executor:
max_turns: 40 # model calls per milestone before it stops as capped
failure_limit: 2 # consecutive failed commands before escalating (A12)
expert:
consult_budget: 3 # ask_expert calls per milestone (A14)
delegation: # hard caps on one delegation (A13)
max_turns: 30
max_tokens: 200000
wall_clock_s: 900
tools:
command_timeout_s: 120 # one run_command before it is killed
max_output_chars: 4000 # tool output kept for the model, head and tail
context: # fitting each request into the model's window
keep_recent_turns: 6 # turns never cut or dropped
safety_margin: 0.1 # fraction of the window held back
nisse-dev/pyproject.toml
+5 -1
View File
@@ -3,7 +3,11 @@ name = "nisse-dev"
version = "0.1.0"
description = "Dev agent: plans and implements software projects, produces Gitea PRs"
requires-python = ">=3.12"
dependencies = ["nisse-core", "httpx>=0.27"]
dependencies = [
"nisse-core",
"httpx>=0.27",
"pyyaml>=6",
]
[project.scripts]
nisse-dev = "nisse_dev.main:main"
nisse-dev/src/nisse_dev/agent.py
+291
View File
@@ -0,0 +1,291 @@
"""Composition root: the one place that knows how the whole agent fits together.
Every component below has been unit-tested in isolation for some time; nothing
had ever assembled them. That gap hid two live bugs — the delegation harness
and the Gitea comment poller were both fully built and reachable from nothing
but their own tests — which is the argument for having a single root rather
than constructing pieces ad hoc.
"""
import threading
from collections.abc import Callable
from dataclasses import dataclass
from nisse_dev.approvals import PlanBoard
from nisse_dev.cancel import Cancel
from nisse_dev.catalog import ModelCatalog, select_client
from nisse_dev.config import AgentConfig
from nisse_dev.context import BudgetedClient
from nisse_dev.control import Orchestrator
from nisse_dev.delegation import Caps, Delegation
from nisse_dev.executor import LLMExecutor
from nisse_dev.expert import ExpertDesk
from nisse_dev.git import GitRepo, authenticated_url
from nisse_dev.gitea import Comment, GiteaClient
from nisse_dev.llm import AnthropicPassthroughClient, LiteLLMClient
from nisse_dev.matrix import MatrixTransport
from nisse_dev.oplog import OpLog
from nisse_dev.planner import LLMPlanningAgent
from nisse_dev.poller import CommentPoller
from nisse_dev.publish import MilestonePublisher
from nisse_dev.state import StateStore
from nisse_dev.tools import Workspace
from nisse_dev.worker import SerialWorker
# How long shutdown waits for running work before leaving the store to the
# durability layer (A2/A3). A milestone can run far longer than a service
# manager allows for a stop, and a hard exit mid-work is already safe.
SHUTDOWN_WAIT_S = 10.0
TOOL_DRIVING_ROLES = ("planner", "executor", "delegate")
def discover_models(config: AgentConfig, oplog: OpLog) -> ModelCatalog:
"""Resolve every configured role against LiteLLM before anything runs.
Refusing to boot is the whole point: a role pointed at a model this key
cannot see is a misconfiguration, and the cheapest place to learn that is
here rather than ten minutes into a milestone. An unreachable proxy is
reported separately -- it is an outage, not a mistake, and telling them
apart is what lets an operator know whether to edit config or wait.
"""
catalog = ModelCatalog.discover(config.llm_base_url, config.llm_token)
catalog.validate(config.roles.pinned())
# Every role's context window must be known before a request is built
# against it (A10): a guess that is too large reaches a server that may
# silently cut the prompt.
catalog.require_limits(config.roles.pinned())
for role in TOOL_DRIVING_ROLES:
facts = catalog.facts(config.roles.resolve(role))
if facts.supports_tools is False:
# A warning, not a refusal: models LiteLLM declares tool-less are
# observed driving tool loops anyway, and A15 expects weak local
# models to be accommodated rather than excluded.
oplog.emit(
"model_tools_unsupported",
cause="startup",
role=role,
model=facts.alias,
backing=facts.backing,
)
return catalog
@dataclass
class Agent:
config: AgentConfig
store: StateStore
transport: MatrixTransport
orchestrator: Orchestrator
gitea: GiteaClient
account: str
worker: SerialWorker
_stopped: threading.Event
def start(self) -> None:
"""Boot in the order A3 requires: reconcile against external truth
before accepting a single operator command."""
self.worker.call(self.orchestrator.start)
self.transport.start()
state = self.worker.call(self.store.get_state)
where = state.state if state else "IDLE"
self.transport.send_message(
f"nisse-dev up as {self.account} — project {self.config.project}, "
f"state {where}, planner {self.config.roles.planner}, "
f"executor {self.config.roles.executor}"
)
def wait(self) -> None:
self._stopped.wait()
def stop(self) -> None:
"""Shut down outermost-first: stop accepting work, then stop talking,
then close the state store last so nothing writes to a closed file.
The store belongs to the worker, so the worker closes it."""
self.orchestrator.stop_watching()
self.transport.close()
self.gitea.close()
try:
self.worker.call(self.store.close, timeout=SHUTDOWN_WAIT_S)
except TimeoutError:
pass # still working; exiting mid-work is what A2/A3 are for
self.worker.stop(timeout=SHUTDOWN_WAIT_S)
self._stopped.set()
def make_llm_clients(
config: AgentConfig, catalog: ModelCatalog
) -> tuple[BudgetedClient, BudgetedClient]:
"""The two LLM routes, as every model call in the agent uses them.
Shared with the benchmark, so that what it measures is exactly what the
agent would do: the same context budget, the same declared limits, and
tool_choice asked only of models declared to enforce it.
"""
tuning = config.tuning
def limits_for(model: str) -> tuple[int, int | None]:
facts = catalog.facts(model)
assert facts.max_input_tokens is not None # require_limits checked
return facts.max_input_tokens, facts.max_output_tokens
def budgeted(client: object) -> BudgetedClient:
# Every request, on either route, is fitted to its model's window.
return BudgetedClient(
client,
limits_for,
max_output_tokens=config.max_output_tokens,
keep_recent_turns=tuning.context.keep_recent_turns,
safety_margin=tuning.context.safety_margin,
enforces_tool_choice=lambda m: (
catalog.facts(m).supports_tool_choice is True
),
)
normalized = budgeted(
LiteLLMClient(
config.llm_base_url, config.llm_token, max_tokens=config.max_output_tokens
)
)
# Delegations prefer the vendor-native route; fidelity is cheap
# elsewhere (A5). Which route a given model gets is decided by
# select_client, not by where the caller happens to sit.
passthrough = budgeted(
AnthropicPassthroughClient(
config.llm_base_url, config.llm_token, max_tokens=config.max_output_tokens
)
)
return normalized, passthrough
def prepare_workspace(config: AgentConfig, account: str) -> GitRepo:
"""Ensure the workspace is a clone of the project repo.
Cloning here rather than on `use <project>` is the single-project
shortcut: resolving a project name to a repo is issue #27's work.
"""
repo = GitRepo(config.workspace)
remote = authenticated_url(
config.gitea_url, config.gitea_repo, account, config.gitea_token
)
if not (config.workspace / ".git").exists():
config.workspace.parent.mkdir(parents=True, exist_ok=True)
GitRepo(config.workspace.parent).clone(remote, config.workspace)
repo.configure_identity(account, f"{account}@users.noreply.local")
return repo
def build_agent(
config: AgentConfig, *, notify: Callable[[str], None] | None = None
) -> Agent:
oplog = OpLog("nisse-dev")
worker = SerialWorker(
on_error=lambda exc: oplog.emit(
"worker.job_failed", cause="unhandled exception", error=repr(exc)
)
)
worker.start()
# Opened on the worker, which is the only thread allowed to use it.
store = worker.call(lambda: StateStore(config.state_db, oplog=oplog))
cancel = Cancel()
# Before the clone, before the token checks: a typo'd role name should
# cost a second, not a repository.
catalog = discover_models(config, oplog)
gitea = GiteaClient(config.gitea_url, config.gitea_token)
# Whoami rather than another config variable: it names the identity every
# Gitea permission is evaluated against, and proves the token works before
# anything depends on it.
account = gitea.whoami()
git_repo = prepare_workspace(config, account)
tuning = config.tuning
workspace = Workspace(
config.workspace,
timeout_s=tuning.tools.command_timeout_s,
max_output_chars=tuning.tools.max_output_chars,
)
normalized, passthrough = make_llm_clients(config, catalog)
transport = MatrixTransport(config.matrix)
announce = notify or transport.send_message
expert = ExpertDesk(
normalized, store, config.roles, budget=tuning.expert.consult_budget
)
delegate_model = config.roles.resolve("delegate")
delegation = Delegation(
select_client(delegate_model, normalized, passthrough, catalog),
store,
workspace,
git_repo,
model=delegate_model,
backing_model=catalog.facts(delegate_model).backing,
caps=Caps(
turns=tuning.delegation.max_turns,
tokens=tuning.delegation.max_tokens,
wall_clock_s=tuning.delegation.wall_clock_s,
),
cancel=cancel,
)
executor = LLMExecutor(
normalized,
store,
config.roles,
workspace,
expert=expert,
delegate=delegation,
notify=announce,
cancel=cancel,
max_turns=tuning.executor.max_turns,
failure_limit=tuning.executor.failure_limit,
)
publisher = MilestonePublisher(
store,
gitea,
config.gitea_repo,
git_repo,
base_url=config.gitea_url,
account=account,
token=config.gitea_token,
base_branch=config.base_branch,
)
def poller_factory(on_comment: Callable[[Comment], None]) -> CommentPoller:
return CommentPoller(
gitea,
config.gitea_repo,
on_comment,
oplog=oplog,
# Its own comments are not new input; without this the agent
# answers itself.
ignore_authors={account},
)
orchestrator = Orchestrator(
store,
transport,
LLMPlanningAgent(normalized, store, config.roles, cancel=cancel),
operators=config.operators,
roles=config.roles,
executor=executor,
publisher=publisher,
gitea_operators=config.gitea_operators,
board=PlanBoard(gitea, config.gitea_repo),
delegate=delegation,
poller_factory=poller_factory,
catalog=catalog,
worker=worker,
cancel=cancel,
)
return Agent(
config=config,
store=store,
transport=transport,
orchestrator=orchestrator,
gitea=gitea,
account=account,
worker=worker,
_stopped=threading.Event(),
)
nisse-dev/src/nisse_dev/bench/__init__.py
+17
View File
@@ -0,0 +1,17 @@
"""The executor benchmark: canned milestones, run repeatedly per model.
Choosing an executor model, and later tuning its knobs, should rest on
measured success rates, not on one run or on reputation. Each task is a fixed
approved plan over a seed project, with a hidden check the model never sees;
success means the check passes, not that the model said it finished. Runs use
the agent's own client wiring (context budget, declared limits, tool_choice
only where declared), so what is measured is what the agent would do.
Results are stored, one JSON line per run, with everything needed to compare
them later: the model's identity and declared facts, a note on how it was
served, the task's content hash, the agent's commit, and its tuning. A new
model then needs only its own runs to be compared with everything measured
before it.
See scripts/benchmark.py, and docs/Tuning-Guide.md for how to read results.
"""
nisse-dev/src/nisse_dev/bench/results.py
+165
View File
@@ -0,0 +1,165 @@
"""Stored benchmark results, and the report that compares them.
Every run is one JSON line. The report is rebuilt from all of them each
time, so a model benchmarked today is read alongside every model measured
before it, as long as the task is still the same version.
"""
import json
import math
import statistics
from collections import defaultdict
from collections.abc import Iterable
from dataclasses import asdict, dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from nisse_dev.bench.runner import RunRecord
from nisse_dev.config import Tuning
def results_file(directory: Path, model: str, when: datetime | None = None) -> Path:
day = (when or datetime.now(UTC)).strftime("%Y-%m-%d")
return directory / f"{day}-{model}.jsonl"
def append(path: Path, record: RunRecord) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("a") as out:
out.write(json.dumps(record.to_json(), sort_keys=True) + "\n")
def load(directory: Path) -> list[dict[str, Any]]:
records = []
for path in sorted(directory.glob("*.jsonl")):
for line in path.read_text().splitlines():
if line.strip():
records.append(json.loads(line))
return records
def wilson(successes: int, runs: int, z: float = 1.96) -> tuple[float, float]:
"""A 95% confidence interval for a success rate, sound at small counts.
Three successes in five runs and five in five overlap here, which is the
point: small samples of a noisy model do not separate much."""
if runs == 0:
return 0.0, 1.0
p = successes / runs
denominator = 1 + z * z / runs
centre = (p + z * z / (2 * runs)) / denominator
half = z * math.sqrt(p * (1 - p) / runs + z * z / (4 * runs * runs)) / denominator
return max(0.0, centre - half), min(1.0, centre + half)
@dataclass
class Row:
model: str
note: str
task: str
runs: int
successes: int
median_turns: float
mean_wall_s: float # working time: rate-limit waits are left out
cost_per_run: float
stale: bool
rate_limited: int = 0 # runs that waited out the proxy's rate limit
tuning: str = "" # how the runs' tuning differs from today's defaults
@property
def interval(self) -> tuple[float, float]:
return wilson(self.successes, self.runs)
def tuning_label(tuning: dict[str, Any]) -> str:
"""The settings in a run's tuning that differ from today's defaults, as
"executor.max_turns=24"; empty when there are none. Relative to today's
defaults so that rows at the current settings stay unlabelled."""
defaults = asdict(Tuning())
return ", ".join(
f"{section}.{key}={value}"
for section, values in sorted(tuning.items())
for key, value in sorted((values or {}).items())
if defaults.get(section, {}).get(key) != value
)
def summarise(
records: Iterable[dict[str, Any]], current_hashes: dict[str, str]
) -> list[Row]:
"""One row per (model, how it was served, task, task version, tuning).
Tuning is part of the key because a knob changes the result: runs capped
at 24 turns and runs allowed 40 are not the same measurement."""
groups: dict[tuple[str, str, str, str, str], list[dict[str, Any]]] = defaultdict(
list
)
for r in records:
key = (
r["backing"],
r.get("note", ""),
r["task"],
r["task_hash"],
tuning_label(r.get("tuning", {})),
)
groups[key].append(r)
rows = []
for (backing, note, task, task_hash, tuning), runs in sorted(groups.items()):
rows.append(
Row(
model=backing,
note=note,
task=task,
runs=len(runs),
successes=sum(1 for r in runs if r["success"]),
median_turns=statistics.median(r["turns"] for r in runs),
mean_wall_s=statistics.fmean(
r["wall_s"] - r.get("rate_limit_wait_s", 0.0) for r in runs
),
cost_per_run=statistics.fmean(r["cost"] for r in runs),
stale=current_hashes.get(task) != task_hash,
rate_limited=sum(1 for r in runs if r.get("rate_limit_wait_s")),
tuning=tuning,
)
)
return rows
def render(rows: list[Row]) -> str:
# A tuning column only when some rows need one, so a report at today's
# settings looks as it always has.
tuned = any(row.tuning for row in rows)
header = (
f"{'model':<34} {'note':<22} "
+ (f"{'tuning':<24} " if tuned else "")
+ f"{'task':<15} {'success':>8} {'95% CI':>11} "
f"{'turns':>6} {'time':>7} {'$/run':>8} {'429':>4}"
)
lines = [header, "-" * len(header)]
for row in rows:
low, high = row.interval
task = row.task + (" (old)" if row.stale else "")
tuning = f"{row.tuning[:24]:<24} " if tuned else ""
lines.append(
f"{row.model[:34]:<34} {row.note[:22]:<22} {tuning}{task[:15]:<15} "
f"{row.successes:>3}/{row.runs:<4} {low:>4.0%}-{high:<5.0%} "
f"{row.median_turns:>6.0f} {row.mean_wall_s:>6.0f}s {row.cost_per_run:>8.4f} "
f"{row.rate_limited or '':>4}"
)
if any(row.rate_limited for row in rows):
lines.append("")
lines.append(
"429 = runs that waited out the proxy's rate limit; time leaves the "
"waits out."
)
if tuned:
lines.append("")
lines.append("tuning = settings that differ from today's defaults.")
if any(row.stale for row in rows):
lines.append("")
lines.append(
"(old) = measured against an earlier version of the task; "
"not comparable with current runs."
)
return "\n".join(lines)
nisse-dev/src/nisse_dev/bench/runner.py
+229
View File
@@ -0,0 +1,229 @@
"""Running one task once with one executor model, and recording it."""
import io
import subprocess
import tempfile
import time
from dataclasses import asdict, dataclass, field
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from nisse_dev.bench.tasks import Task
from nisse_dev.canonical import (
CanonicalEvent,
PlanSubmitted,
StoredEvent,
ToolCall,
ToolResult,
UserMessage,
)
from nisse_dev.catalog import ModelFacts
from nisse_dev.config import Tuning
from nisse_dev.executor import LLMExecutor
from nisse_dev.llm import RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
from nisse_dev.tools import Workspace
class MeteringClient:
"""Counts what a run spends, and notices a fallback: an answer from a
model other than the one asked for is not a measurement of it."""
def __init__(self, inner: Any, facts: ModelFacts) -> None:
self._inner = inner
self._facts = facts
self.calls = 0
self.prose_turns = 0
self.tokens_in = 0
self.tokens_out = 0
self.cost = 0.0
self.fallbacks: list[str] = []
self.rate_limit_wait_s = 0.0
def complete(
self,
events: list[StoredEvent],
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
) -> list[CanonicalEvent]:
self.calls += 1
try:
produced = self._inner.complete(events, tools, model, system=system)
finally:
# Counted even when the call fails: a rate limit that never
# cleared still cost the run its wait.
self.rate_limit_wait_s += float(
getattr(self._inner, "last_rate_limit_wait_s", 0.0) or 0.0
)
usage = getattr(self._inner, "last_usage", {}) or {}
self.tokens_in += int(usage.get("input", 0))
self.tokens_out += int(usage.get("output", 0))
self.cost += float(getattr(self._inner, "last_cost", 0.0) or 0.0)
answered = getattr(self._inner, "last_model", None)
if not self._facts.answered(answered):
self.fallbacks.append(str(answered))
if not any(isinstance(e, ToolCall) for e in produced):
self.prose_turns += 1
return produced
@dataclass
class RunRecord:
"""One run, with everything needed to compare it with runs made later
against other models, other servers, or other tuning."""
ts: str
model: str
backing: str
provider: str
model_info: dict[str, Any]
note: str
task: str
task_hash: str
commit: str
tuning: dict[str, Any]
success: bool
outcome: str
outcome_note: str
turns: int
prose_turns: int
failed_commands: int
invented_tools: int
trimmed_requests: int
tokens_in: int
tokens_out: int
cost: float
wall_s: float
fallbacks: list[str] = field(default_factory=list)
# Part of wall_s spent waiting out the proxy's rate limit, not working.
rate_limit_wait_s: float = 0.0
check_output: str = ""
def to_json(self) -> dict[str, Any]:
return asdict(self)
def agent_commit(repo: Path | None = None) -> str:
"""The agent's commit, marked when the tree had uncommitted changes."""
where = repo or Path(__file__).resolve().parent
try:
sha = subprocess.run(
["git", "rev-parse", "--short", "HEAD"],
cwd=where,
capture_output=True,
text=True,
check=True,
).stdout.strip()
dirty = subprocess.run(
["git", "status", "--porcelain"],
cwd=where,
capture_output=True,
text=True,
check=True,
).stdout.strip()
except (OSError, subprocess.CalledProcessError):
return "unknown"
return f"{sha}+dirty" if dirty else sha
def _count(events: list[StoredEvent]) -> tuple[int, int]:
tools = {
e.event.call_id: e.event.tool for e in events if isinstance(e.event, ToolCall)
}
failed = invented = 0
for stored in events:
event = stored.event
if isinstance(event, ToolResult) and not event.success:
if event.result.startswith("no such tool"):
invented += 1
elif tools.get(event.call_id) == "run_command":
failed += 1
return failed, invented
def run_task(
task: Task,
facts: ModelFacts,
client: Any,
tuning: Tuning,
*,
note: str = "",
commit: str = "",
workdir: Path | None = None,
) -> RunRecord:
"""Run one task once: seed a workspace, hand the executor the approved
plan, then judge the result with the hidden check.
Delegation and ask_expert are off: an escalation is a failure of the
model under test, not a frontier model's success."""
with tempfile.TemporaryDirectory(dir=workdir) as tmp:
root = Path(tmp)
workspace = Workspace(
root / "work",
timeout_s=tuning.tools.command_timeout_s,
max_output_chars=tuning.tools.max_output_chars,
)
task.seed_into(workspace.root)
store = StateStore(root / "state.db", oplog=OpLog("nisse-bench", io.StringIO()))
store.append_event(PlanSubmitted(plan=task.plan, model="benchmark"))
store.append_event(UserMessage(text="start milestone 1", source="benchmark"))
meter = MeteringClient(client, facts)
trims_before = getattr(client, "trimmed_requests", 0)
model = facts.alias
roles = RoleMap(executor=model, planner=model, delegate=model, consultant=model)
executor = LLMExecutor(
meter,
store,
roles,
workspace,
max_turns=tuning.executor.max_turns,
failure_limit=tuning.executor.failure_limit,
)
started = time.monotonic()
try:
outcome = executor.run(1)
status, detail = outcome.status, outcome.note
except Exception as exc: # noqa: BLE001 -- recorded, the benchmark goes on
status, detail = "error", repr(exc)
wall = time.monotonic() - started
success, check_output = task.run_check(workspace.root)
failed, invented = _count(store.read_events())
store.close()
return RunRecord(
ts=datetime.now(UTC).isoformat(timespec="seconds"),
model=model,
backing=facts.backing,
provider=facts.provider,
model_info={
"max_input_tokens": facts.max_input_tokens,
"max_output_tokens": facts.max_output_tokens,
"supports_tool_choice": facts.supports_tool_choice,
"supports_function_calling": facts.supports_tools,
},
note=note,
task=task.name,
task_hash=task.content_hash,
commit=commit,
tuning=asdict(tuning),
success=success and not meter.fallbacks,
outcome=status,
outcome_note=detail[:500],
turns=meter.calls,
prose_turns=meter.prose_turns,
failed_commands=failed,
invented_tools=invented,
trimmed_requests=getattr(client, "trimmed_requests", 0) - trims_before,
tokens_in=meter.tokens_in,
tokens_out=meter.tokens_out,
cost=round(meter.cost, 6),
wall_s=round(wall, 1),
fallbacks=meter.fallbacks,
rate_limit_wait_s=meter.rate_limit_wait_s,
check_output=check_output,
)
nisse-dev/src/nisse_dev/bench/tasks.py
+102
View File
@@ -0,0 +1,102 @@
"""Benchmark tasks: a seed project, an approved plan, and a hidden check.
Each task is a directory under ``tasks/``:
- ``task.yaml``: ``title``, ``plan`` (the approved milestone plan, written the
way the planner is asked to write them), and ``check`` (a shell command run
in the workspace afterwards).
- ``seed/``: the project the executor starts from.
- ``hidden/``: files the check needs, copied in only after the run, into
``.bench_check/``. The executor never sees them.
- ``solution/``: a reference solution, overlaid on the seed by the tests to
prove each task can be solved and that its check tells the difference.
"""
import hashlib
import shutil
import subprocess
from dataclasses import dataclass
from pathlib import Path
import yaml
TASKS_DIR = Path(__file__).parent / "tasks"
CHECK_DIR = ".bench_check"
@dataclass(frozen=True)
class Task:
name: str
title: str
plan: str
check: str
path: Path
@property
def content_hash(self) -> str:
"""What the task is, so a changed task is never compared with runs of
its earlier version. Covers everything the model or the check sees;
not the reference solution."""
digest = hashlib.sha256()
for part in ("task.yaml", "seed", "hidden"):
root = self.path / part
files = [root] if root.is_file() else sorted(root.rglob("*"))
for file in files:
if file.is_file() and "__pycache__" not in file.parts:
digest.update(str(file.relative_to(self.path)).encode())
digest.update(file.read_bytes())
return digest.hexdigest()[:16]
def seed_into(self, workspace: Path) -> None:
shutil.copytree(self.path / "seed", workspace, dirs_exist_ok=True)
def apply_solution(self, workspace: Path) -> None:
shutil.copytree(self.path / "solution", workspace, dirs_exist_ok=True)
def run_check(self, workspace: Path, timeout_s: int = 300) -> tuple[bool, str]:
"""Copy the hidden files in and run the check. Returns whether it
passed, and its output for the record."""
target = workspace / CHECK_DIR
if target.exists():
shutil.rmtree(target) # never trust a directory the model made
shutil.copytree(self.path / "hidden", target)
try:
result = subprocess.run(
self.check,
shell=True,
cwd=workspace,
capture_output=True,
text=True,
timeout=timeout_s,
check=False,
)
except subprocess.TimeoutExpired:
return False, f"check timed out after {timeout_s}s"
output = (result.stdout + result.stderr)[-2000:]
return result.returncode == 0, output
def load_task(path: Path) -> Task:
spec = yaml.safe_load((path / "task.yaml").read_text())
return Task(
name=path.name,
title=str(spec["title"]),
plan=str(spec["plan"]).strip(),
check=str(spec["check"]),
path=path,
)
def load_tasks(names: list[str] | None = None, root: Path = TASKS_DIR) -> list[Task]:
available = {
p.name: p for p in sorted(root.iterdir()) if (p / "task.yaml").exists()
}
if names:
unknown = [n for n in names if n not in available]
if unknown:
raise ValueError(
f"unknown task(s): {', '.join(unknown)}; "
f"available: {', '.join(available)}"
)
return [load_task(available[n]) for n in names]
return [load_task(p) for p in available.values()]
nisse-dev/src/nisse_dev/bench/tasks/fix-bug/hidden/original_test_inventory.py.txt
+11
View File
@@ -0,0 +1,11 @@
from inventory import Item, low_stock, total_value
def test_total_value_counts_every_unit() -> None:
items = [Item("bolt", 0.25, 100), Item("nut", 0.10, 40)]
assert total_value(items) == 29.0
def test_low_stock_is_alphabetical() -> None:
items = [Item("washer", 0.05, 2), Item("bolt", 0.25, 100), Item("anchor", 3.0, 1)]
assert low_stock(items) == ["anchor", "washer"]
nisse-dev/src/nisse_dev/bench/tasks/fix-bug/hidden/test_hidden.py
+27
View File
@@ -0,0 +1,27 @@
from pathlib import Path
import pytest
from inventory import Item, low_stock, total_value
def test_the_visible_tests_were_not_changed() -> None:
original = (Path(__file__).parent / "original_test_inventory.py.txt").read_text()
assert Path("test_inventory.py").read_text() == original
@pytest.mark.parametrize(
("items", "expected"),
[
([], 0.0),
([Item("bolt", 0.25, 100), Item("nut", 0.10, 40)], 29.0),
([Item("gear", 12.5, 3)], 37.5),
([Item("empty", 9.99, 0)], 0.0),
],
)
def test_total_value(items: list[Item], expected: float) -> None:
assert total_value(items) == pytest.approx(expected)
def test_low_stock_still_works() -> None:
items = [Item("washer", 0.05, 2), Item("bolt", 0.25, 100)]
assert low_stock(items) == ["washer"]
nisse-dev/src/nisse_dev/bench/tasks/fix-bug/seed/inventory.py
+23
View File
@@ -0,0 +1,23 @@
"""A tiny inventory."""
from dataclasses import dataclass
@dataclass
class Item:
name: str
unit_price: float
quantity: int
def total_value(items: list[Item]) -> float:
"""The value of all stock: each item's unit price times its quantity."""
total = 0.0
for item in items:
total += item.unit_price
return total
def low_stock(items: list[Item], threshold: int = 5) -> list[str]:
"""Names of items with fewer than `threshold` units, alphabetically."""
return sorted(item.name for item in items if item.quantity < threshold)
nisse-dev/src/nisse_dev/bench/tasks/fix-bug/seed/test_inventory.py
+11
View File
@@ -0,0 +1,11 @@
from inventory import Item, low_stock, total_value
def test_total_value_counts_every_unit() -> None:
items = [Item("bolt", 0.25, 100), Item("nut", 0.10, 40)]
assert total_value(items) == 29.0
def test_low_stock_is_alphabetical() -> None:
items = [Item("washer", 0.05, 2), Item("bolt", 0.25, 100), Item("anchor", 3.0, 1)]
assert low_stock(items) == ["anchor", "washer"]
nisse-dev/src/nisse_dev/bench/tasks/fix-bug/solution/inventory.py
+23
View File
@@ -0,0 +1,23 @@
"""A tiny inventory."""
from dataclasses import dataclass
@dataclass
class Item:
name: str
unit_price: float
quantity: int
def total_value(items: list[Item]) -> float:
"""The value of all stock: each item's unit price times its quantity."""
total = 0.0
for item in items:
total += item.unit_price * item.quantity
return total
def low_stock(items: list[Item], threshold: int = 5) -> list[str]:
"""Names of items with fewer than `threshold` units, alphabetically."""
return sorted(item.name for item in items if item.quantity < threshold)
nisse-dev/src/nisse_dev/bench/tasks/fix-bug/task.yaml
+12
View File
@@ -0,0 +1,12 @@
title: Fix a bug a failing test exposes
check: python -m pytest -q .bench_check
plan: |
Milestone 1: fix the failing inventory test.
1. Run `python -m pytest -q` and read the failure.
2. The failing test is correct; the bug is in `inventory.py`. Do not change
any test file.
3. Fix `total_value` so that each item contributes its unit price times its
quantity.
4. Run `python -m pytest -q` until every test passes.
5. Call finish.
nisse-dev/src/nisse_dev/bench/tasks/long-output/hidden/test_hidden.py
+11
View File
@@ -0,0 +1,11 @@
import pytest
from rates import RATES
def test_every_region_is_present_once() -> None:
assert sorted(RATES) == [f"R{n:03d}" for n in range(300)]
@pytest.mark.parametrize("n", range(300))
def test_rate(n: int) -> None:
assert RATES[f"R{n:03d}"] == pytest.approx(n % 25 / 100)
nisse-dev/src/nisse_dev/bench/tasks/long-output/seed/rates.py
+304
View File
@@ -0,0 +1,304 @@
"""Tax rate per region code."""
RATES = {
"R000": 0.00,
"R001": 0.01,
"R002": 0.02,
"R003": 0.03,
"R004": 0.04,
"R005": 0.05,
"R006": 0.06,
"R007": 0.07,
"R008": 0.08,
"R009": 0.09,
"R010": 0.10,
"R011": 0.11,
"R012": 0.12,
"R013": 0.13,
"R014": 0.14,
"R015": 0.15,
"R016": 0.16,
"R017": 0.17,
"R018": 0.18,
"R019": 0.19,
"R020": 0.20,
"R021": 0.21,
"R022": 0.22,
"R023": 0.23,
"R024": 0.24,
"R025": 0.00,
"R026": 0.01,
"R027": 0.02,
"R028": 0.03,
"R029": 0.04,
"R030": 0.05,
"R031": 0.06,
"R032": 0.07,
"R033": 0.08,
"R034": 0.09,
"R035": 0.10,
"R036": 0.11,
"R037": 0.12,
"R038": 0.13,
"R039": 0.14,
"R040": 0.15,
"R041": 0.16,
"R042": 0.17,
"R043": 0.18,
"R044": 0.19,
"R045": 0.20,
"R046": 0.21,
"R047": 0.22,
"R048": 0.23,
"R049": 0.24,
"R050": 0.00,
"R051": 0.01,
"R052": 0.02,
"R053": 0.03,
"R054": 0.04,
"R055": 0.05,
"R056": 0.06,
"R057": 0.07,
"R058": 0.08,
"R059": 0.09,
"R060": 0.10,
"R061": 0.11,
"R062": 0.12,
"R063": 0.13,
"R064": 0.14,
"R065": 0.15,
"R066": 0.16,
"R067": 0.17,
"R068": 0.18,
"R069": 0.19,
"R070": 0.20,
"R071": 0.21,
"R072": 0.22,
"R073": 0.23,
"R074": 0.24,
"R075": 0.00,
"R076": 0.01,
"R077": 0.02,
"R078": 0.03,
"R079": 0.04,
"R080": 0.05,
"R081": 0.06,
"R082": 0.07,
"R083": 0.08,
"R084": 0.09,
"R085": 0.10,
"R086": 0.11,
"R087": 0.12,
"R088": 0.13,
"R089": 0.14,
"R090": 0.15,
"R091": 0.16,
"R092": 0.17,
"R093": 0.18,
"R094": 0.19,
"R095": 0.20,
"R096": 0.21,
"R097": 0.22,
"R098": 0.23,
"R099": 0.24,
"R100": 0.00,
"R101": 0.01,
"R102": 0.02,
"R103": 0.03,
"R104": 0.04,
"R105": 0.05,
"R106": 0.06,
"R107": 0.07,
"R108": 0.08,
"R109": 0.09,
"R110": 0.10,
"R111": 0.11,
"R112": 0.12,
"R113": 0.13,
"R114": 0.14,
"R115": 0.15,
"R116": 0.16,
"R117": 0.17,
"R118": 0.18,
"R119": 0.19,
"R120": 0.20,
"R121": 0.21,
"R122": 0.22,
"R123": 0.23,
"R124": 0.24,
"R125": 0.00,
"R126": 0.01,
"R127": 0.02,
"R128": 0.03,
"R129": 0.04,
"R130": 0.05,
"R131": 0.06,
"R132": 0.07,
"R133": 0.08,
"R134": 0.09,
"R135": 0.10,
"R136": 0.11,
"R137": 0.12,
"R138": 0.13,
"R139": 0.14,
"R140": 0.15,
"R141": 0.16,
"R142": 0.17,
"R143": 0.18,
"R144": 0.19,
"R145": 0.20,
"R146": 0.21,
"R147": 0.22,
"R148": 0.23,
"R149": 0.24,
"R150": 0.00,
"R151": 0.01,
"R152": 0.02,
"R153": 0.03,
"R154": 0.04,
"R155": 0.05,
"R156": 0.06,
"R157": 0.07,
"R158": 0.08,
"R159": 0.09,
"R160": 0.10,
"R161": 0.11,
"R162": 0.12,
"R163": 0.13,
"R164": 0.14,
"R165": 0.15,
"R166": 0.16,
"R167": 0.17,
"R168": 0.18,
"R169": 0.19,
"R170": 0.20,
"R171": 0.21,
"R172": 0.22,
"R173": 0.23,
"R174": 0.24,
"R175": 0.00,
"R176": 0.01,
"R177": 0.02,
"R178": 0.03,
"R179": 0.04,
"R180": 0.05,
"R181": 0.06,
"R182": 0.07,
"R183": 0.08,
"R184": 0.09,
"R185": 0.10,
"R186": 0.11,
"R187": 0.12,
"R188": 0.13,
"R189": 0.14,
"R190": 0.15,
"R191": 0.16,
"R192": 0.17,
"R193": 0.18,
"R194": 0.19,
"R195": 0.20,
"R196": 0.21,
"R197": 0.22,
"R198": 0.23,
"R199": 0.24,
"R200": 0.00,
"R201": 0.01,
"R202": 0.02,
"R203": 0.03,
"R204": 0.04,
"R205": 0.05,
"R206": 0.06,
"R207": 0.07,
"R208": 0.08,
"R209": 0.09,
"R210": 0.10,
"R211": 0.11,
"R212": 0.12,
"R213": 0.13,
"R214": 0.14,
"R215": 0.15,
"R216": 0.16,
"R217": 0.17,
"R218": 0.18,
"R219": 0.19,
"R220": 0.20,
"R221": 0.21,
"R222": 0.22,
"R223": 0.23,
"R224": 0.24,
"R225": 0.00,
"R226": 0.01,
"R227": 0.02,
"R228": 0.03,
"R229": 0.04,
"R230": 0.05,
"R231": 0.06,
"R232": 0.07,
"R233": 0.08,
"R234": 0.09,
"R235": 0.10,
"R236": 0.11,
"R237": 0.21,
"R238": 0.13,
"R239": 0.14,
"R240": 0.15,
"R241": 0.16,
"R242": 0.17,
"R243": 0.18,
"R244": 0.19,
"R245": 0.20,
"R246": 0.21,
"R247": 0.22,
"R248": 0.23,
"R249": 0.24,
"R250": 0.00,
"R251": 0.01,
"R252": 0.02,
"R253": 0.03,
"R254": 0.04,
"R255": 0.05,
"R256": 0.06,
"R257": 0.07,
"R258": 0.08,
"R259": 0.09,
"R260": 0.10,
"R261": 0.11,
"R262": 0.12,
"R263": 0.13,
"R264": 0.14,
"R265": 0.15,
"R266": 0.16,
"R267": 0.17,
"R268": 0.18,
"R269": 0.19,
"R270": 0.20,
"R271": 0.21,
"R272": 0.22,
"R273": 0.23,
"R274": 0.24,
"R275": 0.00,
"R276": 0.01,
"R277": 0.02,
"R278": 0.03,
"R279": 0.04,
"R280": 0.05,
"R281": 0.06,
"R282": 0.07,
"R283": 0.08,
"R284": 0.09,
"R285": 0.10,
"R286": 0.11,
"R287": 0.12,
"R288": 0.13,
"R289": 0.14,
"R290": 0.15,
"R291": 0.16,
"R292": 0.17,
"R293": 0.18,
"R294": 0.19,
"R295": 0.20,
"R296": 0.21,
"R297": 0.22,
"R298": 0.23,
"R299": 0.24,
}
nisse-dev/src/nisse_dev/bench/tasks/long-output/seed/test_rates.py
+7
View File
@@ -0,0 +1,7 @@
import pytest
from rates import RATES
@pytest.mark.parametrize("code", [f"R{n:03d}" for n in range(300)])
def test_rate(code: str) -> None:
assert RATES[code] == pytest.approx(int(code[1:]) % 25 / 100)
nisse-dev/src/nisse_dev/bench/tasks/long-output/solution/rates.py
+304
View File
@@ -0,0 +1,304 @@
"""Tax rate per region code."""
RATES = {
"R000": 0.00,
"R001": 0.01,
"R002": 0.02,
"R003": 0.03,
"R004": 0.04,
"R005": 0.05,
"R006": 0.06,
"R007": 0.07,
"R008": 0.08,
"R009": 0.09,
"R010": 0.10,
"R011": 0.11,
"R012": 0.12,
"R013": 0.13,
"R014": 0.14,
"R015": 0.15,
"R016": 0.16,
"R017": 0.17,
"R018": 0.18,
"R019": 0.19,
"R020": 0.20,
"R021": 0.21,
"R022": 0.22,
"R023": 0.23,
"R024": 0.24,
"R025": 0.00,
"R026": 0.01,
"R027": 0.02,
"R028": 0.03,
"R029": 0.04,
"R030": 0.05,
"R031": 0.06,
"R032": 0.07,
"R033": 0.08,
"R034": 0.09,
"R035": 0.10,
"R036": 0.11,
"R037": 0.12,
"R038": 0.13,
"R039": 0.14,
"R040": 0.15,
"R041": 0.16,
"R042": 0.17,
"R043": 0.18,
"R044": 0.19,
"R045": 0.20,
"R046": 0.21,
"R047": 0.22,
"R048": 0.23,
"R049": 0.24,
"R050": 0.00,
"R051": 0.01,
"R052": 0.02,
"R053": 0.03,
"R054": 0.04,
"R055": 0.05,
"R056": 0.06,
"R057": 0.07,
"R058": 0.08,
"R059": 0.09,
"R060": 0.10,
"R061": 0.11,
"R062": 0.12,
"R063": 0.13,
"R064": 0.14,
"R065": 0.15,
"R066": 0.16,
"R067": 0.17,
"R068": 0.18,
"R069": 0.19,
"R070": 0.20,
"R071": 0.21,
"R072": 0.22,
"R073": 0.23,
"R074": 0.24,
"R075": 0.00,
"R076": 0.01,
"R077": 0.02,
"R078": 0.03,
"R079": 0.04,
"R080": 0.05,
"R081": 0.06,
"R082": 0.07,
"R083": 0.08,
"R084": 0.09,
"R085": 0.10,
"R086": 0.11,
"R087": 0.12,
"R088": 0.13,
"R089": 0.14,
"R090": 0.15,
"R091": 0.16,
"R092": 0.17,
"R093": 0.18,
"R094": 0.19,
"R095": 0.20,
"R096": 0.21,
"R097": 0.22,
"R098": 0.23,
"R099": 0.24,
"R100": 0.00,
"R101": 0.01,
"R102": 0.02,
"R103": 0.03,
"R104": 0.04,
"R105": 0.05,
"R106": 0.06,
"R107": 0.07,
"R108": 0.08,
"R109": 0.09,
"R110": 0.10,
"R111": 0.11,
"R112": 0.12,
"R113": 0.13,
"R114": 0.14,
"R115": 0.15,
"R116": 0.16,
"R117": 0.17,
"R118": 0.18,
"R119": 0.19,
"R120": 0.20,
"R121": 0.21,
"R122": 0.22,
"R123": 0.23,
"R124": 0.24,
"R125": 0.00,
"R126": 0.01,
"R127": 0.02,
"R128": 0.03,
"R129": 0.04,
"R130": 0.05,
"R131": 0.06,
"R132": 0.07,
"R133": 0.08,
"R134": 0.09,
"R135": 0.10,
"R136": 0.11,
"R137": 0.12,
"R138": 0.13,
"R139": 0.14,
"R140": 0.15,
"R141": 0.16,
"R142": 0.17,
"R143": 0.18,
"R144": 0.19,
"R145": 0.20,
"R146": 0.21,
"R147": 0.22,
"R148": 0.23,
"R149": 0.24,
"R150": 0.00,
"R151": 0.01,
"R152": 0.02,
"R153": 0.03,
"R154": 0.04,
"R155": 0.05,
"R156": 0.06,
"R157": 0.07,
"R158": 0.08,
"R159": 0.09,
"R160": 0.10,
"R161": 0.11,
"R162": 0.12,
"R163": 0.13,
"R164": 0.14,
"R165": 0.15,
"R166": 0.16,
"R167": 0.17,
"R168": 0.18,
"R169": 0.19,
"R170": 0.20,
"R171": 0.21,
"R172": 0.22,
"R173": 0.23,
"R174": 0.24,
"R175": 0.00,
"R176": 0.01,
"R177": 0.02,
"R178": 0.03,
"R179": 0.04,
"R180": 0.05,
"R181": 0.06,
"R182": 0.07,
"R183": 0.08,
"R184": 0.09,
"R185": 0.10,
"R186": 0.11,
"R187": 0.12,
"R188": 0.13,
"R189": 0.14,
"R190": 0.15,
"R191": 0.16,
"R192": 0.17,
"R193": 0.18,
"R194": 0.19,
"R195": 0.20,
"R196": 0.21,
"R197": 0.22,
"R198": 0.23,
"R199": 0.24,
"R200": 0.00,
"R201": 0.01,
"R202": 0.02,
"R203": 0.03,
"R204": 0.04,
"R205": 0.05,
"R206": 0.06,
"R207": 0.07,
"R208": 0.08,
"R209": 0.09,
"R210": 0.10,
"R211": 0.11,
"R212": 0.12,
"R213": 0.13,
"R214": 0.14,
"R215": 0.15,
"R216": 0.16,
"R217": 0.17,
"R218": 0.18,
"R219": 0.19,
"R220": 0.20,
"R221": 0.21,
"R222": 0.22,
"R223": 0.23,
"R224": 0.24,
"R225": 0.00,
"R226": 0.01,
"R227": 0.02,
"R228": 0.03,
"R229": 0.04,
"R230": 0.05,
"R231": 0.06,
"R232": 0.07,
"R233": 0.08,
"R234": 0.09,
"R235": 0.10,
"R236": 0.11,
"R237": 0.12,
"R238": 0.13,
"R239": 0.14,
"R240": 0.15,
"R241": 0.16,
"R242": 0.17,
"R243": 0.18,
"R244": 0.19,
"R245": 0.20,
"R246": 0.21,
"R247": 0.22,
"R248": 0.23,
"R249": 0.24,
"R250": 0.00,
"R251": 0.01,
"R252": 0.02,
"R253": 0.03,
"R254": 0.04,
"R255": 0.05,
"R256": 0.06,
"R257": 0.07,
"R258": 0.08,
"R259": 0.09,
"R260": 0.10,
"R261": 0.11,
"R262": 0.12,
"R263": 0.13,
"R264": 0.14,
"R265": 0.15,
"R266": 0.16,
"R267": 0.17,
"R268": 0.18,
"R269": 0.19,
"R270": 0.20,
"R271": 0.21,
"R272": 0.22,
"R273": 0.23,
"R274": 0.24,
"R275": 0.00,
"R276": 0.01,
"R277": 0.02,
"R278": 0.03,
"R279": 0.04,
"R280": 0.05,
"R281": 0.06,
"R282": 0.07,
"R283": 0.08,
"R284": 0.09,
"R285": 0.10,
"R286": 0.11,
"R287": 0.12,
"R288": 0.13,
"R289": 0.14,
"R290": 0.15,
"R291": 0.16,
"R292": 0.17,
"R293": 0.18,
"R294": 0.19,
"R295": 0.20,
"R296": 0.21,
"R297": 0.22,
"R298": 0.23,
"R299": 0.24,
}
nisse-dev/src/nisse_dev/bench/tasks/long-output/task.yaml
+12
View File
@@ -0,0 +1,12 @@
title: Find one failure in long test output
check: python -m pytest -q .bench_check
plan: |
Milestone 1: fix the one wrong entry in the rate table.
1. Run `python -m pytest -v`. Exactly one test fails, and its output names a
region code.
2. The tests are correct. In `rates.py`, the rate for region code RNNN must be
(NNN mod 25) / 100. Correct only the failing region's entry; leave every
other line as it is.
3. Run `python -m pytest -q` until every test passes.
4. Call finish.
nisse-dev/src/nisse_dev/bench/tasks/multi-file/hidden/test_hidden.py
+33
View File
@@ -0,0 +1,33 @@
import subprocess
import sys
from greeter.cli import main
from greeter.core import greet
def test_default_is_unchanged() -> None:
assert greet("Bob") == "Hello, Bob!"
def test_shout() -> None:
assert greet("Bob", shout=True) == "HELLO, BOB!"
def test_cli_default(capsys) -> None:
assert main(["Ada"]) == 0
assert capsys.readouterr().out == "Hello, Ada!\n"
def test_cli_shout(capsys) -> None:
assert main(["--shout", "Ada"]) == 0
assert capsys.readouterr().out == "HELLO, ADA!\n"
def test_module_entry_point() -> None:
result = subprocess.run(
[sys.executable, "-m", "greeter", "--shout", "Bob"],
capture_output=True,
text=True,
check=True,
)
assert result.stdout == "HELLO, BOB!\n"
nisse-dev/src/nisse_dev/bench/tasks/multi-file/seed/greeter/__init__.py
View File
Whitespace-only changes.
nisse-dev/src/nisse_dev/bench/tasks/multi-file/seed/greeter/__main__.py
+3
View File
@@ -0,0 +1,3 @@
from greeter.cli import main
raise SystemExit(main())
nisse-dev/src/nisse_dev/bench/tasks/multi-file/seed/greeter/cli.py
+11
View File
@@ -0,0 +1,11 @@
import argparse
from greeter.core import greet
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(prog="greeter")
parser.add_argument("name")
args = parser.parse_args(argv)
print(greet(args.name))
return 0
nisse-dev/src/nisse_dev/bench/tasks/multi-file/seed/greeter/core.py
+2
View File
@@ -0,0 +1,2 @@
def greet(name: str) -> str:
return f"Hello, {name}!"
nisse-dev/src/nisse_dev/bench/tasks/multi-file/seed/tests/test_greeter.py
+11
View File
@@ -0,0 +1,11 @@
from greeter.cli import main
from greeter.core import greet
def test_greet() -> None:
assert greet("Bob") == "Hello, Bob!"
def test_cli_prints_the_greeting(capsys) -> None:
assert main(["Bob"]) == 0
assert capsys.readouterr().out == "Hello, Bob!\n"
nisse-dev/src/nisse_dev/bench/tasks/multi-file/solution/greeter/cli.py
+12
View File
@@ -0,0 +1,12 @@
import argparse
from greeter.core import greet
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(prog="greeter")
parser.add_argument("name")
parser.add_argument("--shout", action="store_true")
args = parser.parse_args(argv)
print(greet(args.name, shout=args.shout))
return 0
nisse-dev/src/nisse_dev/bench/tasks/multi-file/solution/greeter/core.py
+3
View File
@@ -0,0 +1,3 @@
def greet(name: str, shout: bool = False) -> str:
greeting = f"Hello, {name}!"
return greeting.upper() if shout else greeting
nisse-dev/src/nisse_dev/bench/tasks/multi-file/task.yaml
+16
View File
@@ -0,0 +1,16 @@
title: Thread a new option through several files
check: python -m pytest -q .bench_check
plan: |
Milestone 1: add a --shout option to the greeter.
1. In `greeter/core.py`, give `greet` a keyword argument `shout: bool = False`.
When it is true, return the whole greeting in upper case:
greet("Bob", shout=True) == "HELLO, BOB!". The default behaviour is unchanged.
2. In `greeter/cli.py`, add a `--shout` flag (action="store_true") and pass it
to `greet`.
3. Add two tests to `tests/test_greeter.py`: one for greet with shout=True, and
one that calls `main(["--shout", "Bob"])` and checks the printed output
using pytest's capsys fixture.
4. Run `python -m pytest -q` until every test passes, and check that
`python -m greeter --shout Bob` prints `HELLO, BOB!`.
5. Call finish.
nisse-dev/src/nisse_dev/bench/tasks/write-function/hidden/test_hidden.py
+19
View File
@@ -0,0 +1,19 @@
import pytest
from slugify import slugify
@pytest.mark.parametrize(
("text", "expected"),
[
("Hello, World!", "hello-world"),
(" Nisse Agent ", "nisse-agent"),
("a--b", "a-b"),
("", ""),
("Python 3.12 Release", "python-3-12-release"),
("--leading and trailing--", "leading-and-trailing"),
("MiXeD CaSe", "mixed-case"),
("!!!", ""),
],
)
def test_slugify(text: str, expected: str) -> None:
assert slugify(text) == expected
nisse-dev/src/nisse_dev/bench/tasks/write-function/seed/README.md
+1
View File
@@ -0,0 +1 @@
A small utilities project.
nisse-dev/src/nisse_dev/bench/tasks/write-function/solution/slugify.py
+5
View File
@@ -0,0 +1,5 @@
import re
def slugify(text: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")
nisse-dev/src/nisse_dev/bench/tasks/write-function/task.yaml
+13
View File
@@ -0,0 +1,13 @@
title: Write a small function and its tests
check: python -m pytest -q .bench_check
plan: |
Milestone 1: add a slugify function.
1. Create `slugify.py` in the workspace root, defining `slugify(text: str) -> str`.
2. It lowercases the text, replaces every run of characters that are not a-z or
0-9 with a single hyphen, and strips hyphens from both ends. Examples:
"Hello, World!" -> "hello-world"; " Nisse Agent " -> "nisse-agent";
"a--b" -> "a-b"; "" -> "".
3. Create `test_slugify.py` with pytest tests covering the four examples.
4. Run `python -m pytest -q test_slugify.py`, and fix anything until it passes.
5. Call finish.
nisse-dev/src/nisse_dev/cancel.py
+40
View File
@@ -0,0 +1,40 @@
"""The operator's kill switch, as the working loops see it (spec A4).
`stop` has to reach work that is already running, from a thread that is not
running it. So it is a flag, set by whoever hears the command and checked by
the loops at each tool boundary. Setting it touches nothing durable: the
record of a stop is written by the worker when the stopped work returns.
"""
import threading
from nisse_dev.canonical import ToolCall, ToolResult
from nisse_dev.state import StateStore
STOPPED_RESULT = "not run: the operator stopped the agent"
class Cancel:
def __init__(self) -> None:
self._event = threading.Event()
def set(self) -> None:
self._event.set()
def clear(self) -> None:
self._event.clear()
@property
def requested(self) -> bool:
return self._event.is_set()
def skip_calls(
store: StateStore, calls: list[ToolCall], reason: str = STOPPED_RESULT
) -> None:
"""Record calls left unrun. Every tool call needs a result for the
transcript to render again, and the result says what happened."""
for call in calls:
store.append_event(
ToolResult(call_id=call.call_id, result=reason, success=False)
)
nisse-dev/src/nisse_dev/canonical.py
+5 -1
View File
@@ -56,8 +56,12 @@ class GateOpened:
class DelegationStarted:
brief: str
role: str
model: str
model: str # the alias asked for, e.g. "remote-architecture"
caps: dict[str, Any] # turns / tokens / wall-clock limits
# What the alias resolved to at task start. An alias is free to change
# models between tasks, so eval history (A16) keyed on the alias alone
# would average results across whatever it happened to point at.
backing_model: str = ""
@dataclass(frozen=True)
nisse-dev/src/nisse_dev/catalog.py
+227
View File
@@ -0,0 +1,227 @@
"""Runtime model discovery (spec A5, A11, A13).
LiteLLM is the VM's single LLM destination, which makes it the authority on
what any given model *is*. Asking it at startup keeps vendor knowledge in one
place instead of duplicating it as name patterns in agent code -- and that
stops being a tidiness argument the moment a role alias no longer names its
vendor. An alias like ``remote-architecture`` is deliberately free to point at
Anthropic today and something else tomorrow; nothing in the name can tell us
which request shape the model behind it expects.
Discovery is per key, not global: ``/model/info`` returns only the models the
presented token is allowed to use. So the same check that catches a typo'd
alias also catches a role pointed at a model this key cannot reach.
"""
from collections.abc import Mapping
from dataclasses import dataclass
from typing import Any
import httpx
from nisse_dev.llm import LLMClient, LLMError
# LiteLLM's provider name for models we have a vendor-native adapter for.
ANTHROPIC_PROVIDER = "anthropic"
class CatalogUnavailable(LLMError):
"""The catalog could not be fetched: the proxy is down, unreachable, or
refused the token. An outage, not a mistake in the configuration."""
class UnknownModel(LLMError):
"""A model was named that this key's catalog does not contain: a typo, a
stale alias, or a key without access. Always a misconfiguration."""
class UnknownLimits(LLMError):
"""A model in use has no declared context window. LiteLLM knows the
limits of commercial models; for a local one they are whatever its server
allocates, and have to be declared in LiteLLM's ``model_info``. Guessing
is not safe: a server that silently truncates a prompt (Ollama does)
drops its start, which is where the system prompt and the plan are."""
@dataclass(frozen=True)
class ModelFacts:
alias: str # what we ask for, e.g. "remote-architecture"
backing: str # what LiteLLM calls, e.g. "anthropic/claude-opus-5"
provider: str # "anthropic", "openai", "ollama", ...
supports_tools: bool | None # None when LiteLLM does not say
# The whole context window, prompt plus response, as LiteLLM declares it
# (model_info.max_input_tokens); None when it does not say.
max_input_tokens: int | None = None
# The most one response may use (model_info.max_output_tokens).
max_output_tokens: int | None = None
# Whether its server enforces tool_choice (model_info.supports_tool_choice).
# Only a declared True counts: a model that says nothing is not asked.
supports_tool_choice: bool | None = None
@property
def native_anthropic(self) -> bool:
return self.provider == ANTHROPIC_PROVIDER
def answered(self, reported: str | None) -> bool:
"""Whether a response reporting this model name came from this model.
A proxy fallback answers under another model's name. LiteLLM reports
the alias; a provider may report its own name for the backing model,
sometimes with a date suffix ("claude-haiku-4-5-20251001").
"""
if not reported:
return True # nothing reported, nothing contradicted
own_name = self.backing.partition("/")[2] or self.backing
return reported in (self.alias, self.backing) or reported.startswith(own_name)
class ModelCatalog:
"""What one LiteLLM key can see, as of the moment it was asked."""
def __init__(self, facts: Mapping[str, ModelFacts]) -> None:
self._facts = dict(facts)
@classmethod
def from_payload(cls, payload: Mapping[str, Any]) -> "ModelCatalog":
facts: dict[str, ModelFacts] = {}
for entry in payload.get("data") or []:
alias = entry.get("model_name")
if not alias:
continue
params = entry.get("litellm_params") or {}
info = entry.get("model_info") or {}
backing = str(params.get("model") or "")
# litellm_provider is the direct answer; the backing model's
# "vendor/model" prefix says the same thing and is present even
# when the provider field is not.
provider = info.get("litellm_provider") or backing.partition("/")[0]
facts[alias] = ModelFacts(
alias=alias,
backing=backing,
provider=str(provider),
supports_tools=info.get("supports_function_calling"),
max_input_tokens=_positive_int(info.get("max_input_tokens")),
max_output_tokens=_positive_int(info.get("max_output_tokens")),
supports_tool_choice=info.get("supports_tool_choice"),
)
return cls(facts)
@classmethod
def discover(
cls,
base_url: str,
token: str,
*,
timeout_s: float = 30.0,
transport: httpx.BaseTransport | None = None,
) -> "ModelCatalog":
try:
with httpx.Client(
base_url=base_url.rstrip("/"),
headers={"Authorization": f"Bearer {token}"},
timeout=timeout_s,
transport=transport,
) as http:
response = http.get("/model/info")
except httpx.HTTPError as exc:
raise CatalogUnavailable(
f"could not reach the LiteLLM proxy at {base_url} to discover "
f"models: {exc}"
) from exc
if response.status_code != 200:
raise CatalogUnavailable(
f"the LiteLLM proxy refused a model listing with "
f"{response.status_code}: {response.text[:300]}"
)
return cls.from_payload(response.json())
def aliases(self) -> list[str]:
return sorted(self._facts)
def facts(self, alias: str) -> ModelFacts:
try:
return self._facts[alias]
except KeyError:
raise UnknownModel(
f"{alias!r} is not a model this LiteLLM key can see; "
f"available: {', '.join(self.aliases()) or '(none)'}"
) from None
def validate(self, roles: Mapping[str, str]) -> None:
"""Check a whole role mapping at once, before anything uses it.
Reported together rather than one per boot: an operator fixing a
config wants the whole list, not to rediscover the next typo on the
next start (the same reason ``AgentConfig`` reports every missing
setting at once).
"""
unknown = {
role: alias for role, alias in roles.items() if alias not in self._facts
}
if unknown:
named = ", ".join(
f"{role}={alias!r}" for role, alias in sorted(unknown.items())
)
raise UnknownModel(
f"role(s) point at models this LiteLLM key cannot see: {named}; "
f"available: {', '.join(self.aliases()) or '(none)'}"
)
def require_limits(self, roles: Mapping[str, str]) -> None:
"""Refuse a role mapping that uses a model with no declared context
window, naming every such model at once."""
missing = {
role: alias
for role, alias in roles.items()
if self.facts(alias).max_input_tokens is None
}
if missing:
named = ", ".join(
f"{role}={alias!r}" for role, alias in sorted(missing.items())
)
raise UnknownLimits(
f"no context window is declared for: {named}. Set "
"model_info.max_input_tokens (and max_output_tokens) for these "
"in LiteLLM; docs/Tuning-Guide.md says how to find the values"
)
def pinned(self, roles: Mapping[str, str]) -> dict[str, dict[str, Any]]:
"""The resolved mapping to record at task start (A11, A16).
Recording only the alias would let an alias that silently changes
models muddle the eval history behind it: "remote-architecture: 80%
good" averaged over however many models it pointed at is not a fact
about anything.
"""
return {
role: {
"alias": alias,
"backing": self.facts(alias).backing,
"provider": self.facts(alias).provider,
}
for role, alias in roles.items()
}
def _positive_int(value: Any) -> int | None:
if isinstance(value, bool) or not isinstance(value, int | float):
return None
return int(value) if value > 0 else None
def select_client(
model: str,
normalized: LLMClient,
passthrough: LLMClient,
catalog: ModelCatalog,
) -> LLMClient:
"""Choose the route from what the model *is*, not from what it is called.
A13 spells out the rule: use the vendor-native adapter where one exists,
and otherwise provisionally accept LiteLLM's normalization for that
vendor -- validating it against the A16 eval set before trusting it with
delegations. There is deliberately no name-based fallback here: a second
heuristic could disagree with the proxy, and an unknown model raises
rather than quietly taking the normalized route.
"""
return passthrough if catalog.facts(model).native_anthropic else normalized
nisse-dev/src/nisse_dev/config.py
+365
View File
@@ -0,0 +1,365 @@
"""Agent configuration (spec A5: nothing hardcoded).
Configuration is a YAML file: roles, connections, and the tuning knobs (see
docs/Tuning-Guide.md). It holds no secrets. A credential is written as a
reference, ``token: ${NISSE_LLM_TOKEN}``, and resolved from the environment,
so the file can be reviewed, shared, or committed without leaking. The
environment supplies those references and nothing else: every setting has
exactly one home.
Every value is collected and validated **before** anything is built, and a
single error names every problem at once. The alternative — failing at first
use — surfaces a missing token twenty minutes into a milestone, on the call
that mattered. Unknown keys are errors too: a misspelt tuning knob that was
silently ignored would look exactly like a knob that has no effect.
"""
import os
import re
from collections.abc import Mapping
from dataclasses import dataclass, field, fields
from pathlib import Path
from typing import Any
import yaml
from nisse_dev.llm import DEFAULT_MAX_OUTPUT_TOKENS, RoleMap
from nisse_dev.matrix import MatrixConfig
DEFAULT_CONFIG_FILE = "nisse-dev.yaml"
REQUIRED_KEYS = (
"state_db",
"workspace",
"project",
"llm.base_url",
"llm.token",
"roles.executor",
"roles.planner",
"roles.delegate",
"roles.consultant",
"matrix.homeserver",
"matrix.user_id",
"matrix.token",
"matrix.room_id",
"matrix.operators",
"gitea.url",
"gitea.token",
"gitea.repo",
"gitea.operators",
)
OPTIONAL_KEYS = ("llm.max_output_tokens", "gitea.base_branch")
_REFERENCE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}")
class ConfigError(RuntimeError):
pass
# --- tuning ----------------------------------------------------------------
# The defaults are the values the agent ran with before they were
# configurable, so an absent `tuning:` section changes nothing.
@dataclass(frozen=True)
class ExecutorTuning:
max_turns: int = 40
failure_limit: int = 2
@dataclass(frozen=True)
class ExpertTuning:
consult_budget: int = 3
@dataclass(frozen=True)
class DelegationTuning:
max_turns: int = 30
max_tokens: int = 200_000
wall_clock_s: float = 900.0
@dataclass(frozen=True)
class ToolsTuning:
command_timeout_s: int = 120
max_output_chars: int = 4000
@dataclass(frozen=True)
class ContextTuning:
keep_recent_turns: int = 6
safety_margin: float = 0.1
@dataclass(frozen=True)
class Tuning:
executor: ExecutorTuning = field(default_factory=ExecutorTuning)
expert: ExpertTuning = field(default_factory=ExpertTuning)
delegation: DelegationTuning = field(default_factory=DelegationTuning)
tools: ToolsTuning = field(default_factory=ToolsTuning)
context: ContextTuning = field(default_factory=ContextTuning)
_TUNING_SECTIONS: dict[str, type] = {
f.name: f.default_factory # type: ignore[misc]
for f in fields(Tuning)
}
def _parse_tuning(raw: Any, errors: list[str]) -> Tuning:
if raw is None:
return Tuning()
if not isinstance(raw, dict):
errors.append("tuning: must be a mapping")
return Tuning()
sections: dict[str, Any] = {}
for name, section in raw.items():
cls = _TUNING_SECTIONS.get(name)
if cls is None:
errors.append(f"tuning.{name}: unknown section")
continue
if not isinstance(section, dict):
errors.append(f"tuning.{name}: must be a mapping")
continue
known = {f.name: f for f in fields(cls)}
values: dict[str, Any] = {}
for key, value in section.items():
spec = known.get(key)
if spec is None:
errors.append(f"tuning.{name}.{key}: unknown setting")
continue
number = _positive(value, float if spec.type in (float, "float") else int)
if number is None:
errors.append(f"tuning.{name}.{key}: must be a positive number")
continue
values[key] = number
sections[name] = cls(**values)
margin = sections.get("context", ContextTuning()).safety_margin
if margin >= 1:
errors.append("tuning.context.safety_margin: must be less than 1")
return Tuning(**sections)
def _positive(value: Any, kind: type) -> int | float | None:
# bool is an int in Python; `true` for a turn limit is a mistake.
if isinstance(value, bool) or not isinstance(value, int | float):
return None
if kind is int and not float(value).is_integer():
return None
if value <= 0:
return None
return kind(value)
# --- loading ---------------------------------------------------------------
def _people(raw: Any) -> frozenset[str]:
"""Operator allowlists: a YAML list, or comma-separated in the
environment. Blanks are dropped so a trailing comma cannot admit an empty
identity."""
parts = raw if isinstance(raw, list) else str(raw).split(",")
return frozenset(str(part).strip() for part in parts if str(part).strip())
def _flatten(tree: Mapping[str, Any], prefix: str = "") -> dict[str, Any]:
flat: dict[str, Any] = {}
for key, value in tree.items():
dotted = f"{prefix}{key}"
if isinstance(value, dict) and dotted != "tuning":
flat.update(_flatten(value, f"{dotted}."))
else:
flat[dotted] = value
return flat
def _resolve(value: Any, key: str, env: Mapping[str, str], errors: list[str]) -> Any:
"""Replace ${VAR} references with the environment's value. A reference to
an unset variable is an error, never an empty string: an empty token would
pass for a value and fail at the first request."""
if not isinstance(value, str):
return value
def lookup(match: re.Match[str]) -> str:
name = match.group(1)
found = (env.get(name) or "").strip()
if not found:
errors.append(f"{key}: references ${{{name}}}, which is unset or empty")
return found
return _REFERENCE.sub(lookup, value)
def _read_file(path: Path, errors: list[str]) -> dict[str, Any]:
try:
loaded = yaml.safe_load(path.read_text())
except yaml.YAMLError as exc:
errors.append(f"{path}: not valid YAML ({exc})")
return {}
if loaded is None:
return {}
if not isinstance(loaded, dict):
errors.append(f"{path}: must be a mapping at the top level")
return {}
return loaded
@dataclass(frozen=True)
class AgentConfig:
state_db: Path
workspace: Path
project: str
llm_base_url: str
llm_token: str
roles: RoleMap
matrix: MatrixConfig
operators: frozenset[str]
gitea_url: str
gitea_token: str
gitea_repo: str
gitea_operators: frozenset[str]
base_branch: str = "main"
max_output_tokens: int = DEFAULT_MAX_OUTPUT_TOKENS
tuning: Tuning = field(default_factory=Tuning)
config_file: Path | None = None
@classmethod
def load(cls, env: Mapping[str, str] = os.environ) -> "AgentConfig":
"""The file named by NISSE_CONFIG, or ./nisse-dev.yaml."""
path = Path(env.get("NISSE_CONFIG") or DEFAULT_CONFIG_FILE).expanduser()
if not path.is_file():
raise ConfigError(
f"cannot start: no config file at {path}; create it from "
"nisse-dev.example.yaml, or point NISSE_CONFIG at one"
)
return cls.from_file(path, env)
@classmethod
def from_file(
cls, path: Path, env: Mapping[str, str] = os.environ
) -> "AgentConfig":
errors: list[str] = []
tree = _read_file(path, errors)
return cls._build(tree, env, errors, config_file=path)
@classmethod
def from_tree(
cls, tree: Mapping[str, Any], env: Mapping[str, str] = os.environ
) -> "AgentConfig":
"""From an already-parsed config, as the file would give it."""
return cls._build(tree, env, [], config_file=None)
@classmethod
def _build(
cls,
tree: Mapping[str, Any],
env: Mapping[str, str],
errors: list[str],
*,
config_file: Path | None,
) -> "AgentConfig":
tuning = _parse_tuning(tree.get("tuning"), errors)
flat = _flatten(tree)
flat.pop("tuning", None)
known = set(REQUIRED_KEYS) | set(OPTIONAL_KEYS)
for key in sorted(set(flat) - known):
errors.append(f"{key}: unknown setting")
values = {
key: _resolve(value, key, env, errors)
for key, value in flat.items()
if key in known
}
# Whitespace-only counts as missing: a token of spaces would pass a
# truthiness check and fail at the first request instead.
missing = [
key
for key in REQUIRED_KEYS
if values.get(key) is None
or (isinstance(values[key], str) and not values[key].strip())
or (isinstance(values[key], list) and not _people(values[key]))
]
if missing:
errors.append("unset or empty — " + ", ".join(missing))
max_output = values.get("llm.max_output_tokens", DEFAULT_MAX_OUTPUT_TOKENS)
try:
max_output = int(max_output)
if max_output <= 0:
raise ValueError
except (TypeError, ValueError):
errors.append("llm.max_output_tokens: must be a positive integer")
if errors:
raise ConfigError("cannot start: " + "; ".join(errors))
config = cls(
state_db=Path(values["state_db"]).expanduser(),
workspace=Path(values["workspace"]).expanduser(),
project=str(values["project"]),
llm_base_url=str(values["llm.base_url"]),
llm_token=str(values["llm.token"]),
roles=RoleMap(
executor=str(values["roles.executor"]),
planner=str(values["roles.planner"]),
delegate=str(values["roles.delegate"]),
consultant=str(values["roles.consultant"]),
),
matrix=MatrixConfig(
homeserver=str(values["matrix.homeserver"]).rstrip("/"),
user_id=str(values["matrix.user_id"]),
token=str(values["matrix.token"]),
room_id=str(values["matrix.room_id"]),
),
operators=_people(values["matrix.operators"]),
gitea_url=str(values["gitea.url"]),
gitea_token=str(values["gitea.token"]),
gitea_repo=str(values["gitea.repo"]),
gitea_operators=_people(values["gitea.operators"]),
base_branch=str(values.get("gitea.base_branch") or "main"),
max_output_tokens=max_output,
tuning=tuning,
config_file=config_file,
)
config._refuse_secrets_inside_the_clone(env)
return config
def _refuse_secrets_inside_the_clone(self, env: Mapping[str, str]) -> None:
"""The workspace is a git clone the agent commits with ``git add -A``
and pushes. Anything inside it reaches the project repository, so a
state store or an env file placed there would publish credentials as a
side effect of the agent working correctly — silently, and once. The
config file holds no secrets itself, but it names where they come
from and how the agent is deployed, which is not the project's
business either.
"""
workspace = self.workspace.expanduser().resolve()
candidates = [
("state_db", self.state_db),
("NISSE_ENV", Path(env.get("NISSE_ENV", ".env"))),
]
if self.config_file is not None:
candidates.append(("NISSE_CONFIG", self.config_file))
offenders = []
for label, path in candidates:
resolved = path.expanduser().resolve()
if resolved == workspace or workspace in resolved.parents:
offenders.append(f"{label} ({resolved})")
if offenders:
raise ConfigError(
"refusing to start: these sit inside the workspace clone at "
f"{workspace} and would be committed and pushed to the project "
"repository — " + ", ".join(offenders)
)
def redacted(self) -> dict[str, str]:
"""What the agent may safely announce about how it is configured."""
return {
"project": self.project,
"workspace": str(self.workspace),
"repo": self.gitea_repo,
"planner": self.roles.planner,
"executor": self.roles.executor,
"delegate": self.roles.delegate,
}
nisse-dev/src/nisse_dev/context.py
+293
View File
@@ -0,0 +1,293 @@
"""The context budget (spec A10: token budgeting is a render-time concern).
Every model call re-sends a transcript that only grows, and a model server
does not always refuse one that no longer fits: Ollama silently drops the
start of the prompt, which is where the system prompt and the plan are. So a
request is made to fit the model's declared window *before* it is sent, and
one that cannot be made to fit is never sent.
The log itself is never changed. Fitting works on the events about to be
rendered, in three steps, stopping at the first that fits:
1. Tool results older than the most recent turns are cut to their first and
last lines, with a note saying how much was left out.
2. The oldest events are dropped, keeping tool calls with their results, and
a note at the start says how many.
3. If even the recent turns and the system prompt do not fit, the call
fails with ContextOverflow. The caller treats it like any failed call:
the executor stops and asks the operator.
Token counts are estimated from characters, conservatively, and the estimate
is corrected from the input-token counts the server reports back. The
correction only ever makes the estimate larger: underestimating is the
dangerous direction.
"""
import json
from collections.abc import Callable, Sequence
from dataclasses import dataclass, replace
from typing import Any
from nisse_dev.canonical import (
AssistantText,
CanonicalEvent,
DelegationStarted,
PlanSubmitted,
StoredEvent,
ToolCall,
ToolResult,
UserMessage,
)
from nisse_dev.llm import DEFAULT_MAX_OUTPUT_TOKENS, LLMError
# English and code run at about four characters a token. Three is deliberately
# pessimistic, before any correction from what the server reports.
CHARS_PER_TOKEN = 3.0
# Role markers and formatting each rendered message costs beyond its text.
MESSAGE_OVERHEAD_TOKENS = 8
# How much of an older tool result survives step 1.
ELIDED_RESULT_CHARS = 600
# Events the model produced. A turn starts where these start.
_MODEL_SIDE = (AssistantText, ToolCall, PlanSubmitted, DelegationStarted)
class ContextOverflow(LLMError):
"""The request cannot be made to fit the model's context window."""
def head_and_tail(text: str, limit: int) -> str:
"""Keep the start and, mostly, the end of a long text.
The end matters more: test runners and compilers put their summary of
what failed last. The start keeps the exit code and the first lines.
"""
if len(text) <= limit:
return text
head = limit // 4
tail = limit - head
dropped = len(text) - limit
return f"{text[:head]}\n[... {dropped} characters omitted ...]\n{text[-tail:]}"
def _event_chars(event: CanonicalEvent) -> int:
match event:
case ToolCall(tool=tool, arguments=arguments):
return len(tool) + len(json.dumps(arguments))
case ToolResult(result=result):
return len(result)
case _:
text = getattr(event, "text", None) or getattr(event, "plan", None)
if text is None:
text = json.dumps(getattr(event, "__dict__", {}), default=str)
return len(str(text))
def estimate_tokens(
events: Sequence[StoredEvent],
system: str | None,
tools: list[dict[str, Any]] | None,
factor: float = 1.0,
) -> int:
chars = sum(_event_chars(e.event) for e in events)
chars += len(system or "") + len(json.dumps(tools or []))
raw = chars / CHARS_PER_TOKEN + MESSAGE_OVERHEAD_TOKENS * (len(events) + 1)
return int(raw * factor) + 1
def _turn_starts(events: Sequence[StoredEvent]) -> list[int]:
starts = []
previous_model_side = False
for index, stored in enumerate(events):
model_side = isinstance(stored.event, _MODEL_SIDE)
if model_side and not previous_model_side:
starts.append(index)
previous_model_side = model_side
return starts
def _consistent(events: list[StoredEvent]) -> list[StoredEvent]:
"""Drop results whose call is gone: a provider rejects a tool result
with nothing to answer."""
calls = {e.event.call_id for e in events if isinstance(e.event, ToolCall)}
return [
e
for e in events
if not isinstance(e.event, ToolResult) or e.event.call_id in calls
]
@dataclass(frozen=True)
class Budget:
window: int # prompt plus response, in tokens
max_output: int # the ceiling sent with the request
keep_recent_turns: int
safety_margin: float # fraction of the window held back
@property
def for_prompt(self) -> int:
return int(self.window * (1 - self.safety_margin)) - self.max_output
def fit(
events: Sequence[StoredEvent],
system: str | None,
tools: list[dict[str, Any]] | None,
budget: Budget,
factor: float = 1.0,
) -> list[StoredEvent]:
"""The events to render: all of them if they fit, otherwise the result of
the three steps in the module docstring."""
limit = budget.for_prompt
def fits(candidate: Sequence[StoredEvent]) -> bool:
return estimate_tokens(candidate, system, tools, factor) <= limit
kept = list(events)
if fits(kept):
return kept
starts = _turn_starts(kept)
recent_from = (
starts[-budget.keep_recent_turns]
if len(starts) >= budget.keep_recent_turns
else 0
)
# Step 1: cut older tool results down.
kept = [
replace(
stored,
event=replace(
stored.event,
result=head_and_tail(stored.event.result, ELIDED_RESULT_CHARS),
),
)
if index < recent_from and isinstance(stored.event, ToolResult)
else stored
for index, stored in enumerate(kept)
]
if fits(kept):
return kept
# Step 2: drop the oldest events, one turn at a time, never the recent
# ones. The note keeps the transcript starting with a user message, which
# some chat templates require.
cuts = sorted({s for s in starts if 0 < s < recent_from} | {recent_from} - {0})
for cut in cuts:
remaining = _consistent(kept[cut:])
note = StoredEvent(
id=0,
ts="",
event=UserMessage(
text=(
f"[{cut} earlier events omitted to fit the context window; "
"the log still has them]"
),
source="context-budget",
),
)
candidate = [note, *remaining]
if fits(candidate):
return candidate
# Step 3: nothing more can go.
needed = estimate_tokens(kept[recent_from:], system, tools, factor)
raise ContextOverflow(
f"the system prompt and the last {budget.keep_recent_turns} turns need "
f"about {needed} tokens, but only {limit} fit a {budget.window}-token "
"window after the output ceiling and safety margin"
)
class BudgetedClient:
"""An LLM client that fits every request to the model's window.
Wraps a real client (either route) and takes each model's limits from
``limits_for``, which the agent fills from LiteLLM's model catalog.
"""
def __init__(
self,
inner: Any,
limits_for: Callable[[str], tuple[int, int | None]],
*,
max_output_tokens: int = DEFAULT_MAX_OUTPUT_TOKENS,
keep_recent_turns: int = 6,
safety_margin: float = 0.1,
enforces_tool_choice: Callable[[str], bool] = lambda model: False,
) -> None:
self._inner = inner
self._limits_for = limits_for
# Whether a model's server holds it to the offered tools when asked
# (LiteLLM's supports_tool_choice). Asking one that doesn't can be
# worse than not asking: qwen3-14b can't stop, and rambles until the
# output ceiling.
self._enforces_tool_choice = enforces_tool_choice
self._max_output_tokens = max_output_tokens
self._keep_recent_turns = keep_recent_turns
self._safety_margin = safety_margin
# Per model: how much the character estimate undercounted, as learned
# from reported usage. Starts at 1 and only grows.
self._factors: dict[str, float] = {}
self.trimmed_requests = 0
@property
def last_usage(self) -> dict[str, int]:
return getattr(self._inner, "last_usage", {}) or {}
@property
def last_model(self) -> str | None:
return getattr(self._inner, "last_model", None)
@property
def last_cost(self) -> float:
return float(getattr(self._inner, "last_cost", 0.0) or 0.0)
@property
def last_rate_limit_wait_s(self) -> float:
return float(getattr(self._inner, "last_rate_limit_wait_s", 0.0) or 0.0)
def budget_for(self, model: str) -> Budget:
window, model_max_output = self._limits_for(model)
ceiling = self._max_output_tokens
if model_max_output is not None:
ceiling = min(ceiling, model_max_output)
return Budget(
window=window,
max_output=ceiling,
keep_recent_turns=self._keep_recent_turns,
safety_margin=self._safety_margin,
)
def complete(
self,
events: list[StoredEvent],
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
) -> list[CanonicalEvent]:
budget = self.budget_for(model)
factor = self._factors.get(model, 1.0)
fitted = fit(events, system, tools, budget, factor)
# Counted so the benchmark can report how often a model's window was
# the constraint: any request that had to be cut down or shortened.
if fitted != list(events):
self.trimmed_requests += 1
produced = self._inner.complete(
fitted,
tools,
model,
system=system,
max_tokens=budget.max_output,
# Every caller that offers tools needs one called: the executor
# finishes with finish, the planner with submit_plan or
# ask_human, a delegation with hand_back.
require_tool_call=bool(tools) and self._enforces_tool_choice(model),
)
reported = int(self.last_usage.get("input", 0))
if reported > 0:
estimated = estimate_tokens(fitted, system, tools)
observed = reported / max(estimated, 1)
self._factors[model] = max(factor, observed)
return produced
nisse-dev/src/nisse_dev/control.py
+197 -21
View File
@@ -6,14 +6,20 @@ Gating is structural, not advisory: the agent's only actions are ``SubmitPlan``
advancement past a gate happens only here, when an operator approval or command
arrives over transport. Every position change is checkpointed through the state
store before the loop continues, so a restart lands mid-gate, not mid-air.
All of it runs on one worker (nisse_dev.worker), never on the thread that
delivered a message. The one exception is `stop` (A4), which has to reach
work that is already running: the transport's thread sets the cancel flag
directly, and the worker records the stop once the work has returned.
"""
import threading
import uuid
from collections.abc import Iterable, Mapping
from collections.abc import Iterable, Iterator, Mapping
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Any, Protocol
from nisse_dev.cancel import Cancel
from nisse_dev.canonical import GateOpened, PlanSubmitted, UserMessage
from nisse_dev.gates import Approved, Gate, Ignored, PlanRevision, Rejected
from nisse_dev.git import GitError
@@ -22,6 +28,7 @@ from nisse_dev.llm import RoleMap
from nisse_dev.reconcile import Reconciler, reconcile_on_boot
from nisse_dev.state import StateStore
from nisse_dev.transport import InboundMessage, Transport
from nisse_dev.worker import InlineWorker, Worker
IDLE = "IDLE"
PLANNING_ARCH = "PLANNING_ARCH"
@@ -59,6 +66,10 @@ class PlanningError(RuntimeError):
plan or retrying forever."""
class PlanningStopped(RuntimeError):
"""The operator said `stop` (A4) before a planning turn could run."""
@dataclass(frozen=True)
class SubmitPlan:
content: str
@@ -143,6 +154,10 @@ def _fresh_data() -> dict[str, Any]:
"milestone": 1,
"task_id": None,
"roles": None, # A11 mapping, pinned at task start
# What each pinned alias resolved to, discovered from LiteLLM. An
# alias is free to point somewhere else tomorrow, so the transcript
# has to say what actually answered (A11, A16).
"models": None,
"pr": None, # the open PR for the current milestone, once published
# Pointer to the Gitea plan record (Part H): the state store holds
# only this, so gate state can be re-derived from Gitea after a
@@ -163,11 +178,17 @@ class Orchestrator:
publisher: "MilestonePublisher | None" = None,
gitea_operators: Iterable[str] = (),
board: "PlanBoard | None" = None,
delegate: Any = None,
poller_factory: Any = None,
catalog: Any = None,
worker: Worker | None = None,
cancel: Cancel | None = None,
) -> None:
self._store = store
self._transport = transport
self._agent = agent
self._roles = roles
self._catalog = catalog
self._executor = executor
self._publisher = publisher
self._operators = set(operators)
@@ -176,14 +197,37 @@ class Orchestrator:
# the other.
self._gitea_operators = set(gitea_operators)
self._board = board
# Transport and the comment poller deliver on separate threads;
# state transitions must not interleave.
self._lock = threading.RLock()
self._delegate = delegate
# Builds a poller bound to a comment handler; the orchestrator
# owns *when* to watch, which is a state-machine concern, while
# the factory owns how to build one.
self._poller_factory = poller_factory
self._pollers: dict[str, Any] = {}
# Transport and the comment poller deliver on their own threads, and
# every handler runs on the worker, one at a time, so state
# transitions never interleave.
self._worker = worker or InlineWorker()
# Shared with the executor, delegation and planner, which check it
# at each tool boundary.
self._cancel = cancel or Cancel()
# Read from the transport's thread, so plain values only: whether
# long work is running, and the state name for a "busy" reply.
self._working = False
self._state_name = IDLE
# Set by whichever path halted because of a stop, so the stop's own
# job knows whether it stopped anything.
self._halted_by_stop = False
self._data = _fresh_data()
self._project: str | None = None
transport.on_message(self._handle)
def start(self, reconcilers: Mapping[str, Reconciler] | None = None) -> None:
# A3 step 5 runs before the generic sweep: an interrupted delegation is
# resumed or discarded by inspecting the branch and the log, and only a
# genuinely unclean cut is left for the sweep below to gate on.
if self._delegate is not None:
with self._long_work():
self._delegate.recover(self._transport.send_message)
# The publisher owns the external-truth query for its own intents, so
# a configured publisher always contributes its reconciler (A3).
registered: dict[str, Reconciler] = dict(reconcilers or {})
@@ -191,6 +235,8 @@ class Orchestrator:
registered = {**self._publisher.reconcilers(), **registered}
report = reconcile_on_boot(self._store, registered)
state = self._store.get_state()
if state is not None:
self._state_name = state.state
if state is None:
self._transition(IDLE, cause="boot: fresh state")
self._transport.send_message("nisse-dev ready (idle)")
@@ -229,12 +275,68 @@ class Orchestrator:
)
def _handle(self, msg: InboundMessage) -> None:
with self._lock:
self._handle_locked(msg)
def _handle_locked(self, msg: InboundMessage) -> None:
"""Runs on the transport's thread, so it touches nothing durable and
hands everything to the worker."""
if msg.sender not in self._operators:
return
text = msg.text.strip()
if text == "stop":
# Straight to the flag, not behind the work in the queue: the
# work is what it has to reach (A4).
if self._worker.busy:
self._cancel.set()
self._worker.submit(lambda: self._finish_stop(msg))
return
if self._working:
# Queued, it would run after the work, in whatever state the work
# left behind, which is not the state the operator was answering.
self._transport.send_message(
f"busy ({self._state_name}); say 'stop' to halt"
)
self._worker.submit(lambda: self._record(msg))
return
self._worker.submit(lambda: self._handle_now(msg))
def _record(self, msg: InboundMessage) -> None:
self._store.append_event(
UserMessage(text=msg.text.strip(), source=f"transport:{msg.sender}")
)
def _finish_stop(self, msg: InboundMessage) -> None:
"""The worker's half of `stop`. Whatever was running has returned by
now, and has already moved to AWAIT_INPUT if the stop reached it."""
self._record(msg)
halted = self._halted_by_stop
self._halted_by_stop = False
self._cancel.clear()
if not halted:
self._transport.send_message(f"nothing to stop (state={self._state_name})")
@contextmanager
def _long_work(self) -> Iterator[None]:
self._working = True
try:
yield
finally:
self._working = False
def _halt_for_stop(
self, *, what: str, note: str, next_step: str, pending: dict[str, Any]
) -> None:
"""A4: stopped work moves to AWAIT_INPUT, and what happens next is the
operator's call. The prompt asks why, because the model should hear
the reason before it tries again."""
self._halted_by_stop = True
self._await_input(
cause=f"operator: stop ({what})",
prompt=(
f"stopped {what} ({note}). Why did you stop it? Your reply is "
f"recorded for the model. {next_step}"
),
pending=pending,
)
def _handle_now(self, msg: InboundMessage) -> None:
# Every operator message enters the canonical transcript, so planning
# turns see feedback and answers as conversation rather than as
# out-of-band parameters (A10).
@@ -274,7 +376,17 @@ class Orchestrator:
# Pin the role mapping now (A11): a mid-task config change must not
# split one task across two planners.
pinned = self._roles.pinned() if self._roles is not None else None
self._data = {**_fresh_data(), "task_id": task_id, "roles": pinned}
resolved = (
self._catalog.pinned(pinned)
if self._catalog is not None and pinned is not None
else None
)
self._data = {
**_fresh_data(),
"task_id": task_id,
"roles": pinned,
"models": resolved,
}
self._project = project
self._store.oplog.task_id = task_id
if self._board is not None:
@@ -298,9 +410,20 @@ class Orchestrator:
) -> None:
assert self._project is not None
try:
action = self._agent.plan(
gate_name, self._project, feedback=feedback, answer=answer
with self._long_work():
action = self._agent.plan(
gate_name, self._project, feedback=feedback, answer=answer
)
except PlanningStopped as exc:
# Answering resumes planning with the answer, so the reason for
# the stop is the first thing the planner hears.
self._halt_for_stop(
what=f"planning {gate_name}",
note=str(exc),
next_step="Planning resumes with your reply.",
pending={"kind": "question", "gate": gate_name},
)
return
except PlanningError as exc:
self._await_input(
cause=f"planning failed: {exc}",
@@ -427,7 +550,19 @@ class Orchestrator:
self._transport.send_message(f"[stub] executing milestone {milestone}")
return
self._transport.send_message(f"executing milestone {milestone}")
outcome = self._executor.run(milestone)
with self._long_work():
outcome = self._executor.run(milestone)
if outcome.status == "stopped":
self._halt_for_stop(
what=f"milestone {milestone}",
note=outcome.note,
next_step=(
f"The plan is still approved; say 'start milestone "
f"{milestone}' to run it again."
),
pending={"kind": "milestone_stopped", "milestone": milestone},
)
return
if outcome.status == "completed":
self._transport.send_message(f"milestone {milestone} done: {outcome.note}")
self._publish_milestone(milestone)
@@ -458,17 +593,21 @@ class Orchestrator:
# durability of the record, not the ability to approve.
return
board_state["comment_id"] = post.comment_id
# Approvals can arrive on the issue as well as over transport (A4),
# so watch it for as long as the gate stands.
self._watch("plan", board_state["issue"], self.handle_plan_comment)
def handle_plan_comment(self, comment: Any) -> None:
"""A comment on the project issue may satisfy an open plan gate (A4).
Approvals bind to a revision here exactly as over transport, and the
first valid approval on either channel wins.
first valid approval on either channel wins. Called from the poller
thread, which waits for the worker so its cursor never moves past a
comment that has not been handled.
"""
with self._lock:
self._handle_plan_comment_locked(comment)
self._worker.call(lambda: self._handle_plan_comment_now(comment))
def _handle_plan_comment_locked(self, comment: Any) -> None:
def _handle_plan_comment_now(self, comment: Any) -> None:
if comment.author not in self._gitea_operators:
return
state = self._store.get_state()
@@ -492,10 +631,9 @@ class Orchestrator:
code, so it is operator-scoped like any other (A4, and the same
reasoning C4 applies to tickets).
"""
with self._lock:
self._handle_review_comment_locked(comment)
self._worker.call(lambda: self._handle_review_comment_now(comment))
def _handle_review_comment_locked(self, comment: Any) -> None:
def _handle_review_comment_now(self, comment: Any) -> None:
if comment.author not in self._gitea_operators:
self._store.oplog.emit(
"gitea.comment_ignored",
@@ -521,7 +659,19 @@ class Orchestrator:
self._store.append_event(
UserMessage(text=comment.body, source=f"gitea:comment:{comment.id}")
)
outcome = self._executor.run(milestone)
with self._long_work():
outcome = self._executor.run(milestone)
if outcome.status == "stopped":
self._halt_for_stop(
what=f"the response to review comment {comment.id}",
note=outcome.note,
next_step=(
f"The milestone {milestone} plan is still approved; say "
f"'start milestone {milestone}' to run it again."
),
pending={"kind": "milestone_stopped", "milestone": milestone},
)
return
if outcome.status != "completed":
self._await_input(
cause=f"review response {outcome.status}",
@@ -579,6 +729,8 @@ class Orchestrator:
"url": result.pull_request.url,
"milestone": milestone,
}
self._unwatch("plan") # the gate is behind us; the PR is the thread now
self._watch("review", result.pull_request.number, self.handle_review_comment)
self._transition(PR_OPEN, cause=f"milestone {milestone} published")
self._transport.send_message(
f"{result.note} — review it, then say 'next milestone' or "
@@ -587,12 +739,14 @@ class Orchestrator:
def _handle_pr_open(self, text: str) -> None:
if text == "next milestone":
self._unwatch("review")
self._data["milestone"] = int(self._data.get("milestone", 1)) + 1
self._data["gate"] = None
self._data["pr"] = None
self._transition(MILESTONE_PLANNING, cause="operator: next milestone")
self._run_planning_turn(f"milestone-{self._data['milestone']}")
elif text == "project complete":
self.stop_watching()
self._data = {**_fresh_data()}
self._project = None
self._store.oplog.task_id = None
@@ -607,6 +761,27 @@ class Orchestrator:
"'project complete'"
)
def _watch(self, kind: str, index: int, handler: Any) -> None:
"""Start watching one Gitea thread, replacing any previous watch of
the same kind. Priming first, so attaching does not replay comments
already addressed."""
if self._poller_factory is None:
return
self._unwatch(kind)
poller = self._poller_factory(handler)
poller.prime(index)
poller.start(index)
self._pollers[kind] = poller
def _unwatch(self, kind: str) -> None:
poller = self._pollers.pop(kind, None)
if poller is not None:
poller.stop()
def stop_watching(self) -> None:
for kind in list(self._pollers):
self._unwatch(kind)
def _await_input(self, *, cause: str, prompt: str, pending: dict[str, Any]) -> None:
self._data["pending"] = pending
self._transition(AWAIT_INPUT, cause=cause)
@@ -659,3 +834,4 @@ class Orchestrator:
self._store.transition(
new_state, cause=cause, project=self._project, data=self._data
)
self._state_name = new_state
nisse-dev/src/nisse_dev/delegation.py
+226 -8
View File
@@ -28,6 +28,7 @@ from collections.abc import Callable
from dataclasses import dataclass, field
from typing import Any, Literal
from nisse_dev.cancel import Cancel, skip_calls
from nisse_dev.canonical import (
AssistantText,
DelegationCompleted,
@@ -36,8 +37,9 @@ from nisse_dev.canonical import (
ToolResult,
)
from nisse_dev.git import GitError, GitRepo
from nisse_dev.llm import LLMClient, LLMError
from nisse_dev.state import StateStore
from nisse_dev.llm import LLMClient, LLMError, UpstreamUnavailable
from nisse_dev.reconcile import Reconciler
from nisse_dev.state import Intent, StateStore
from nisse_dev.tools import EXECUTOR_TOOLS, MUTATING_TOOLS, Workspace
DELEGATION_INTENT = "delegation"
@@ -128,7 +130,7 @@ class Handback:
@dataclass
class DelegationResult:
status: Literal["handback", "capped", "failed"]
status: Literal["handback", "capped", "failed", "interrupted", "stopped"]
note: str
handback: Handback | None = None
branch: str | None = None
@@ -136,11 +138,22 @@ class DelegationResult:
tokens: int = 0
duration_s: float = 0.0
@property
def interrupted(self) -> bool:
"""Stopped by something external rather than by the delegate.
The work on the branch may be perfectly good, so it is preserved and
the intent left pending: the next boot treats it exactly like a crash
mid-delegation (A3) and resumes if the cut was clean. Discarding here
would throw away work because the budget ran out.
"""
return self.status == "interrupted"
@property
def usable(self) -> bool:
"""Only a successful handback may be built upon. A cap breach or a
failure leaves a half-edited tree, which A13 says must gate rather
than be resumed."""
"""Only a successful handback may be built upon. A cap breach, a
failure or an operator's stop leaves a half-edited tree, which A13
says must gate rather than be resumed."""
return self.status == "handback" and self.handback is not None
@@ -153,22 +166,42 @@ class Delegation:
git_repo: GitRepo,
*,
model: str,
backing_model: str = "",
caps: Caps | None = None,
now: Callable[[], float] = time.monotonic,
cancel: Cancel | None = None,
) -> None:
self._client = client
self._store = store
self._workspace = workspace
self._git = git_repo
self._model = model
self._backing_model = backing_model
self._caps = caps or Caps()
self._now = now
# A stopped delegation is discarded like a failed one, not preserved
# for resuming as an interrupted one is: the operator stopped it for
# a reason, and the next step is to talk about that reason.
self._cancel = cancel or Cancel()
def run(
self, *, brief: str, branch: str, role: str = "delegate"
) -> DelegationResult:
started_at = self._now()
base_branch = self._git.current_branch()
# Recorded before the branch exists, so a crash anywhere after this
# leaves a durable statement of what was in flight (A2/A3).
intent_id = self._store.begin_intent(
DELEGATION_INTENT,
{
"branch": branch,
"base": base_branch,
"brief": brief,
"role": role,
"caps": self._caps.as_payload(),
},
cause=f"delegating to {self._model}",
)
try:
# Commit whatever the caller had in flight onto their own branch
# first. Otherwise it would ride along onto the delegation branch
@@ -176,6 +209,7 @@ class Delegation:
self._git.commit_all("Checkpoint before delegating")
self._git.checkout_new_branch(branch)
except GitError as exc:
self._store.abort_intent(intent_id, cause=f"could not branch: {exc}")
return DelegationResult("failed", f"could not branch for delegation: {exc}")
self._store.append_event(
@@ -184,6 +218,7 @@ class Delegation:
role=role,
model=self._model,
caps=self._caps.as_payload(),
backing_model=self._backing_model,
)
)
result = self._loop(brief, branch, started_at)
@@ -197,12 +232,86 @@ class Delegation:
exit_status=result.status,
)
)
if result.interrupted:
# Leave the branch and the intent alone: this delegation is not
# finished, it was cut off, and A3's resume rule is exactly the
# machinery for picking it back up.
return result
if not result.usable:
# Abort discards the branch: the caller must never resume on top
# of a partially-edited tree (A13).
self._abandon(branch, base_branch)
# Settled either way: an aborted delegation is a *finished* one, and
# leaving the intent pending would make the next boot think work was
# still in flight.
self._store.commit_intent(intent_id, cause=f"delegation {result.status}")
return result
def resume(self, intent: Intent) -> DelegationResult:
"""Continue a delegation the process died in the middle of (A3 step 5).
The transcript is already in the canonical log, so the loop re-renders
it and carries on. No new branch and no second ``delegation_started``:
this is the same delegation, which is why its vendor-opaque reasoning
may still be replayed (A10 defines a session as the delegation, not the
process lifetime).
"""
payload = intent.payload
branch = str(payload["branch"])
started_at = self._now()
try:
self._git.checkout_existing_branch(branch)
except GitError as exc:
self._store.commit_intent(
intent.id, cause=f"delegation branch unusable: {exc}"
)
return DelegationResult("failed", f"could not resume on {branch}: {exc}")
result = self._loop(str(payload.get("brief", "")), branch, started_at)
result.duration_s = self._now() - started_at
self._store.append_event(
DelegationCompleted(
handback=result.handback.render() if result.handback else result.note,
artifacts=result.artifacts,
tokens=result.tokens,
duration_s=result.duration_s,
exit_status=result.status,
)
)
if not result.usable:
self._abandon(branch, str(payload.get("base", "main")))
self._store.commit_intent(intent.id, cause=f"delegation {result.status}")
return result
def recover(self, notify: Callable[[str], None]) -> DelegationResult | None:
"""Sweep pending delegation intents at boot (A3 step 5).
Settles what is already finished, discards what never started, resumes
a clean cut — announcing it with a *notice*, not a question, because a
clean cut needs no decision from the operator — and leaves an unclean
one pending so the generic boot sweep gates on it.
"""
for intent in self._store.pending_intents():
if intent.kind != DELEGATION_INTENT:
continue
verdict = delegation_verdict(self._store, self._git, intent)
branch = intent.payload.get("branch")
if verdict == "settled":
self._store.commit_intent(
intent.id, cause="delegation had already completed"
)
elif verdict == "absent":
self._store.abort_intent(
intent.id, cause=f"delegation branch {branch} never created"
)
elif verdict == "resume":
notify(
f"resuming the delegation interrupted on {branch}; "
"the transcript and branch are intact"
)
return self.resume(intent)
return None
def _loop(self, brief: str, branch: str, started_at: float) -> DelegationResult:
system = (
f"{SYSTEM_PROMPT}\n\n{self._workspace.describe()}"
@@ -210,8 +319,17 @@ class Delegation:
)
touched: set[str] = set()
did_work = False
spent = 0
for turn in range(self._caps.turns):
if self._cancel.requested:
return DelegationResult(
"stopped",
f"stopped by the operator before turn {turn + 1}",
branch=branch,
artifacts=sorted(touched),
tokens=spent,
)
elapsed = self._now() - started_at
if elapsed >= self._caps.wall_clock_s:
return DelegationResult(
@@ -220,6 +338,7 @@ class Delegation:
f"after {turn} turns",
branch=branch,
artifacts=sorted(touched),
tokens=spent,
)
try:
produced = self._client.complete(
@@ -228,9 +347,35 @@ class Delegation:
model=self._model,
system=system,
)
except UpstreamUnavailable as exc:
return DelegationResult(
"interrupted",
f"stopped by something upstream, not by the work: {exc}",
branch=branch,
artifacts=sorted(touched),
tokens=spent,
)
except LLMError as exc:
return DelegationResult(
"failed", f"delegate call failed: {exc}", branch=branch
"failed",
f"delegate call failed: {exc}",
branch=branch,
tokens=spent,
)
# Metered from what the provider reported, not estimated. The cap
# is a scope limit (A13), distinct from the money cap A5 keeps on
# the LiteLLM key — but an unbounded delegation can still exhaust
# a daily budget and take external models away until it resets.
usage = getattr(self._client, "last_usage", {}) or {}
spent += int(usage.get("input", 0)) + int(usage.get("output", 0))
if spent >= self._caps.tokens:
return DelegationResult(
"capped",
f"token cap of {self._caps.tokens} reached ({spent} spent) "
f"after {turn + 1} turns",
branch=branch,
artifacts=sorted(touched),
tokens=spent,
)
calls: list[ToolCall] = []
@@ -252,9 +397,19 @@ class Delegation:
"delegate produced neither a tool call nor text",
branch=branch,
artifacts=sorted(touched),
tokens=spent,
)
for call in calls:
for index, call in enumerate(calls):
if self._cancel.requested:
skip_calls(self._store, calls[index:])
return DelegationResult(
"stopped",
f"stopped by the operator during turn {turn + 1}",
branch=branch,
artifacts=sorted(touched),
tokens=spent,
)
if call.tool == "hand_back":
handback = _parse_handback(call.arguments)
if handback is not None and handback.succeeded and not did_work:
@@ -301,6 +456,7 @@ class Delegation:
handback=handback if handback.succeeded else None,
branch=branch,
artifacts=sorted(touched),
tokens=spent,
)
outcome = self._workspace.dispatch(call.tool, call.arguments)
@@ -327,6 +483,7 @@ class Delegation:
f"turn cap of {self._caps.turns} reached without a handback",
branch=branch,
artifacts=sorted(touched),
tokens=spent,
)
def _commit_turn(self, turn: int) -> None:
@@ -361,3 +518,64 @@ def _parse_handback(arguments: dict[str, Any]) -> Handback | None:
next=str(arguments["next"]).strip(),
succeeded=succeeded,
)
def delegation_reconciler(store: StateStore, git_repo: GitRepo) -> Reconciler:
"""A3 step 5: decide whether an interrupted delegation may be resumed.
The judgement is an observable check, never model self-assessment — the
same reasoning as A12. Returns:
- ``True`` the delegation is settled or cleanly resumable
- ``False`` it never really started, so there is nothing to resume
- ``None`` the cut is unclean, so boot gates to the operator
"The branch sits at the expected commit" is approximated by the branch
existing and carrying the delegation's commits: per-turn shas are not
recorded in the log today, so a branch rewritten out from under us would
not be detected. Stated rather than glossed, because the check is weaker
than A3's wording implies.
"""
def check(intent: Intent) -> bool | None:
verdict = delegation_verdict(store, git_repo, intent)
if verdict in ("resume", "settled"):
return True
return False if verdict == "absent" else None
return check
def delegation_verdict(
store: StateStore, git_repo: GitRepo, intent: Intent
) -> Literal["resume", "settled", "absent", "gate"]:
"""The A3 step 5 decision, separated from the Reconciler contract because
boot must *act* on it — resuming is not the same as judging settled."""
branch = intent.payload.get("branch")
if not branch:
return "gate"
if _completed_after_start(store):
return "settled" # it finished; the intent merely outlived its commit
if not git_repo.branch_exists(str(branch)):
return "absent" # the crash landed before any work existed
events = store.read_events()
if not events:
return "gate"
last = events[-1].event
if isinstance(last, ToolCall):
# A call with no recorded result: its effect is unknown and cannot be
# verified by inspection, so gate (A3).
return "gate"
if not isinstance(last, AssistantText | ToolResult):
return "gate"
return "resume"
def _completed_after_start(store: StateStore) -> bool:
"""True when the last delegation event in the log is a completion."""
for stored in reversed(store.read_events()):
if isinstance(stored.event, DelegationCompleted):
return True
if isinstance(stored.event, DelegationStarted):
return False
return False
nisse-dev/src/nisse_dev/evals.py
+88
View File
@@ -0,0 +1,88 @@
"""Delegation and consult evaluation capture (spec A16).
Each ``delegation_started`` / ``delegation_completed`` pair is already a clean
input-output record; this pairs them up and attaches a verdict, so "best model
this week" can be decided by re-running a captured set rather than by benchmark
chatter. Without it, swapping the frontier tier becomes something to avoid,
because the effect is unmeasurable.
Verdicts live outside the canonical log: the log records what happened, a
verdict is a later judgement about it, and judgements get revised.
"""
import json
from dataclasses import asdict, dataclass
from typing import Any
from nisse_dev.canonical import DelegationCompleted, DelegationStarted
from nisse_dev.state import StateStore
UNMARKED = "unmarked"
@dataclass(frozen=True)
class EvalRecord:
event_id: int # of the completion, which is what a verdict attaches to
role: str
model: str
brief: str
caps: dict[str, Any]
handback: str
artifacts: list[str]
tokens: int
duration_s: float
exit_status: str
verdict: str = UNMARKED
note: str = ""
def collect(store: StateStore, after: int = 0) -> list[EvalRecord]:
"""Pair each start with the completion that follows it.
A start with no completion is skipped rather than reported half-formed:
it means the process died mid-delegation, which A3 handles, and an
unfinished record would pollute the eval set.
"""
verdicts = store.outcomes()
records: list[EvalRecord] = []
pending: tuple[int, DelegationStarted] | None = None
for stored in store.read_events(after=after):
if isinstance(stored.event, DelegationStarted):
pending = (stored.id, stored.event)
elif isinstance(stored.event, DelegationCompleted) and pending is not None:
_, started = pending
verdict, note = verdicts.get(stored.id, (UNMARKED, ""))
records.append(
EvalRecord(
event_id=stored.id,
role=started.role,
model=started.model,
brief=started.brief,
caps=started.caps,
handback=stored.event.handback,
artifacts=stored.event.artifacts,
tokens=stored.event.tokens,
duration_s=stored.event.duration_s,
exit_status=stored.event.exit_status,
verdict=verdict,
note=note,
)
)
pending = None
return records
def to_jsonl(records: list[EvalRecord]) -> str:
"""JSON lines, so a captured set can be diffed and re-run per record."""
return "\n".join(json.dumps(asdict(r)) for r in records)
def summarise(records: list[EvalRecord]) -> dict[str, Any]:
by_model: dict[str, dict[str, int]] = {}
for record in records:
bucket = by_model.setdefault(
record.model, {"total": 0, "good": 0, "bad": 0, UNMARKED: 0}
)
bucket["total"] += 1
bucket[record.verdict if record.verdict in ("good", "bad") else UNMARKED] += 1
return by_model
nisse-dev/src/nisse_dev/executor.py
+217 -41
View File
@@ -10,39 +10,65 @@ or finished-enough. Until delegation exists (step 7), an escalation gates to
the human with the failures attached rather than engaging a frontier model.
"""
from collections.abc import Callable
from dataclasses import dataclass
from typing import Literal
from typing import Any, Literal
from nisse_dev.canonical import ToolCall, ToolResult
from nisse_dev.cancel import Cancel, skip_calls
from nisse_dev.canonical import PlanSubmitted, StoredEvent, ToolCall, ToolResult
from nisse_dev.delegation import Delegation
from nisse_dev.expert import ASK_EXPERT_TOOL, BudgetExhausted, ExpertDesk
from nisse_dev.llm import LLMClient, LLMError, RoleMap
from nisse_dev.state import StateStore
from nisse_dev.tools import EXECUTOR_TOOLS, MUTATING_TOOLS, Workspace
from nisse_dev.tools import EXECUTOR_TOOLS, MUTATING_TOOLS, ToolOutcome, Workspace
EXECUTOR_SYSTEM = """You are nisse-dev's executor, doing mechanical work.
Follow the approved plan above exactly. It was written to be executed
Follow the approved plan below exactly. It was written to be executed
verbatim; do not redesign it, expand its scope, or skip steps.
Work by calling tools. Verify your work by running the project's tests.
When every step of the plan is done and the tests pass, reply with a short
plain-text summary and no tool call — that ends the milestone."""
Every reply must be a tool call. Verify your work by running the project's
tests. When every step of the plan is done and the tests pass, call finish
with a short summary — that, and nothing else, ends the milestone."""
DEFAULT_MAX_TURNS = 24
DEFAULT_FAILURE_LIMIT = 2
# A turn with no tool call means "the plan is done". Believing that before
# any real work has happened would record an empty milestone as a success, so
# the claim is challenged once first.
_NUDGE = (
"\n\nYou have not made any change yet, so the plan cannot be complete. "
"Carry out the next step of the plan with a tool call, or run the tests "
"to show the work is already done."
FINISH_TOOL: dict[str, Any] = {
"type": "function",
"function": {
"name": "finish",
"description": (
"End the milestone: every step of the plan is done and the tests pass."
),
"parameters": {
"type": "object",
"properties": {
"summary": {"type": "string", "description": "What was done."}
},
"required": ["summary"],
},
},
}
# Finishing before any real work has happened would record an empty
# milestone as a success, so the claim is refused once first.
_NOT_YET = (
"You have not made any change yet, so the plan cannot be complete. Carry "
"out the next step of the plan, or run the tests to show the work is "
"already done, before calling finish."
)
# A reply in prose breaks the tool contract. A server that enforces
# tool_choice never allows one; on one that doesn't, the model is told once.
_PROSE_NUDGE = (
"\n\nYour last reply was prose. Every reply must be a tool call: carry "
"out the next step, or call finish if the plan is complete."
)
@dataclass(frozen=True)
class ExecutorOutcome:
status: Literal["completed", "capped", "escalate"]
status: Literal["completed", "capped", "escalate", "stopped"]
note: str
@@ -56,6 +82,10 @@ class LLMExecutor:
*,
max_turns: int = DEFAULT_MAX_TURNS,
failure_limit: int = DEFAULT_FAILURE_LIMIT,
expert: ExpertDesk | None = None,
delegate: Delegation | None = None,
notify: Callable[[str], None] | None = None,
cancel: Cancel | None = None,
) -> None:
self._client = client
self._store = store
@@ -63,22 +93,124 @@ class LLMExecutor:
self._workspace = workspace
self._max_turns = max_turns
self._failure_limit = failure_limit
self._expert = expert
self._delegate = delegate
# The loop runs synchronously inside the transport callback, so
# without this the agent is silent for the whole milestone and a
# working run looks exactly like a hung one.
self._notify = notify
self._cancel = cancel or Cancel()
def _report(self, turn: int, calls: list[ToolCall]) -> None:
if self._notify is None:
return
summary = ", ".join(
f"{c.tool}({c.arguments.get('path') or c.arguments.get('command', '')[:40]})"
if c.arguments
else c.tool
for c in calls
)
self._notify(f"turn {turn + 1}: {summary}")
def _escalate(
self, milestone: int, failures: int, last_failure: str
) -> ExecutorOutcome | None:
"""Hand the sub-task to a stronger model, per A12.
Returns None when the delegation handed back usable work and the local
loop should carry on; an ExecutorOutcome when the caller must stop.
"""
note = f"{failures} consecutive command failures; last: {last_failure[:500]}"
if self._delegate is None:
return ExecutorOutcome("escalate", note)
# A12 says the failures go in the brief: the delegate must not have to
# rediscover what the local model already learned the hard way.
brief = (
f"Milestone {milestone} is stuck. The local executor tried and its "
f"commands failed {failures} times in a row.\n\n"
f"Most recent failure:\n{last_failure[:2000]}\n\n"
"The approved plan and the full transcript are in the conversation "
"above. Finish this sub-task, staying inside the approved plan."
)
result = self._delegate.run(
brief=brief, branch=f"delegation/milestone-{milestone}-{failures}"
)
if result.usable:
return None
if result.status == "stopped":
return ExecutorOutcome("stopped", f"during a delegation: {result.note}")
# A13: a cap breach or failure leaves a half-edited tree, so the local
# loop must not resume on top of it.
return ExecutorOutcome(
"escalate", f"{note}; delegation {result.status}: {result.note}"
)
def _tools(self) -> list[dict[str, Any]]:
# A12: ask_expert exists in addition to the observable escalation
# triggers, never as a replacement for them.
if self._expert is None:
return [*EXECUTOR_TOOLS, FINISH_TOOL]
return [*EXECUTOR_TOOLS, ASK_EXPERT_TOOL, FINISH_TOOL]
def _consult(self, arguments: dict[str, Any]) -> ToolOutcome:
assert self._expert is not None
question = str(arguments.get("question", "")).strip()
if not question:
return ToolOutcome("ask_expert needs a question", success=False)
try:
return ToolOutcome(self._expert.ask(question).render(), success=True)
except BudgetExhausted as exc:
# Not an error the model should route around: it is the signal
# that it is circling and should get on with the work or fail.
return ToolOutcome(str(exc), success=False)
except LLMError as exc:
return ToolOutcome(f"the consultant did not answer: {exc}", success=False)
def _scope(self) -> tuple[str | None, int]:
"""The approved plan, and where this milestone's events begin.
The most recent plan in the log is the approved milestone plan: no
plan is submitted between approval and execution, or during review
responses. The executor sees that plan and what came after it, and
not the planning conversation or earlier milestones, which a small
context window cannot afford and the plan already distils.
"""
latest: StoredEvent | None = None
for stored in self._store.read_events():
if isinstance(stored.event, PlanSubmitted):
latest = stored
if latest is None:
return None, 0
assert isinstance(latest.event, PlanSubmitted)
return latest.event.plan, latest.id
def run(self, milestone: int) -> ExecutorOutcome:
model = self._roles.resolve("executor")
plan, after = self._scope()
system = (
f"{EXECUTOR_SYSTEM}\n\n{self._workspace.describe()}"
f"\n\nYou are executing milestone {milestone}."
)
if plan is not None:
# In the system prompt, so the context budget never drops it.
system += f"\n\nThe approved plan:\n{plan}"
tools = self._tools()
offered = {tool["function"]["name"] for tool in tools}
consecutive_failures = 0
did_work = False
challenged = False
finish_refused = False
prose_nudged = False
for _turn in range(self._max_turns):
for turn in range(self._max_turns):
# A4: `stop` takes effect at the next tool boundary. The start of
# a turn is one, and so is the gap between two tool calls below.
if self._cancel.requested:
return ExecutorOutcome("stopped", f"before turn {turn + 1}")
try:
produced = self._client.complete(
self._store.read_events(),
tools=EXECUTOR_TOOLS,
self._store.read_events(after=after),
tools=tools,
model=model,
system=system,
)
@@ -92,25 +224,61 @@ class LLMExecutor:
calls.append(event)
if not calls:
if did_work:
return ExecutorOutcome(
"completed", "executor reported the plan done"
)
if not challenged:
# Claiming completion without having touched anything is
# not evidence of completion; ask once before believing it.
challenged = True
system += _NUDGE
if not prose_nudged:
prose_nudged = True
system += _PROSE_NUDGE
continue
return ExecutorOutcome(
"escalate",
"executor claimed the milestone was done without making any change",
"escalate", "executor kept replying in prose instead of tools"
)
for call in calls:
if call.tool in MUTATING_TOOLS:
did_work = True
outcome = self._workspace.dispatch(call.tool, call.arguments)
self._report(turn, calls)
for index, call in enumerate(calls):
if self._cancel.requested:
skip_calls(self._store, calls[index:])
return ExecutorOutcome("stopped", f"during turn {turn + 1}")
if call.tool == "finish":
if not did_work and not finish_refused:
finish_refused = True
outcome = ToolOutcome(_NOT_YET, success=False)
elif not did_work:
return ExecutorOutcome(
"escalate",
"executor claimed the milestone was done without "
"making any change",
)
else:
self._store.append_event(
ToolResult(
call_id=call.call_id,
result="milestone finished",
success=True,
)
)
skip_calls(
self._store,
calls[index + 1 :],
"not run: finish had already ended the milestone",
)
summary = str(call.arguments.get("summary", "")).strip()
return ExecutorOutcome(
"completed", summary or "executor reported the plan done"
)
elif call.tool not in offered:
# A model inventing tools is not doing the work, and with
# no server enforcement nothing else would stop it, so it
# counts toward escalation like a failed command (A12).
outcome = ToolOutcome(
f"no such tool: {call.tool}; the tools are "
+ ", ".join(sorted(offered)),
success=False,
)
elif call.tool == "ask_expert":
outcome = self._consult(call.arguments)
else:
if call.tool in MUTATING_TOOLS:
did_work = True
outcome = self._workspace.dispatch(call.tool, call.arguments)
self._store.append_event(
ToolResult(
call_id=call.call_id,
@@ -118,18 +286,26 @@ class LLMExecutor:
success=outcome.success,
)
)
# Only commands count toward escalation: a failed read is the
# model looking around, a failed command is work not working.
if call.tool == "run_command":
# Commands and invented tools count toward escalation: a
# failed read is the model looking around, a failed command
# is work not working.
if call.tool == "run_command" or call.tool not in offered:
consecutive_failures = (
0 if outcome.success else consecutive_failures + 1
)
if consecutive_failures >= self._failure_limit:
return ExecutorOutcome(
"escalate",
f"{consecutive_failures} consecutive command failures; "
f"last: {outcome.result[:500]}",
# A12: this is the trigger delegation exists for.
# Escalating to the human is the fallback when there
# is no delegate, not the intended path.
escalation = self._escalate(
milestone,
consecutive_failures,
outcome.result,
)
if escalation is not None:
return escalation
consecutive_failures = 0
did_work = True
return ExecutorOutcome(
"capped", f"hit the {self._max_turns}-turn cap for milestone {milestone}"
nisse-dev/src/nisse_dev/expert.py
+214
View File
@@ -0,0 +1,214 @@
"""`ask_expert` (spec A14).
The one case delegation does not cover: a question needing no repo access —
"which of these two approaches is better", "is this API deprecated". A single
stateless call, a text answer, no session state, no diff application.
Two constraints from A14 shape the implementation:
**Structured output is contracted at the prompt level, not the API level.**
Vendors differ on JSON mode and schema enforcement, so the answer is parsed and
validated against a local schema, with one retry. This runs over LiteLLM's
*normalized* route, which A13 reserves for calls where fidelity is cheap.
**A per-task budget**, so a confused executor cannot spend twenty consults
circling. The count comes from the canonical log rather than a counter in
memory: it survives a restart, and it is observable from outside the model,
which is the same reasoning A12 applies to escalation triggers.
Consults are recorded as ``delegation_started`` / ``delegation_completed`` with
role ``consultant``. That is deliberate — A16 wants a "consult/delegation eval
set", and reusing the composite events gets consult evaluation for free rather
than inventing a parallel record.
"""
import json
import re
from dataclasses import dataclass
from nisse_dev.canonical import (
DelegationCompleted,
DelegationStarted,
StoredEvent,
UserMessage,
)
from nisse_dev.llm import LLMClient, LLMError, RoleMap
from nisse_dev.state import StateStore
CONSULTANT_ROLE = "consultant"
DEFAULT_BUDGET = 3
SYSTEM_PROMPT = """You are a consultant answering one self-contained question.
You have no access to the caller's repository, so answer from the question
alone. If the question cannot be answered without seeing the code, say so
plainly in `answer` rather than guessing.
Reply with a single JSON object and nothing else:
{"answer": "...", "confidence": "high" | "medium" | "low", "caveats": "..."}
Keep `answer` direct and specific. Put what would change your mind, or what you
are assuming, in `caveats`."""
_RETRY_NUDGE = (
"\n\nYour previous reply was not a single valid JSON object with the keys "
"answer, confidence and caveats. Reply with only that object."
)
_CONFIDENCES = {"high", "medium", "low"}
ASK_EXPERT_TOOL: dict = {
"type": "function",
"function": {
"name": "ask_expert",
"description": (
"Ask a stronger model one self-contained question that needs no "
"access to this repository — a design trade-off, or how a library "
"behaves. It cannot see your files. Use sparingly; the budget is "
"small."
),
"parameters": {
"type": "object",
"properties": {"question": {"type": "string"}},
"required": ["question"],
},
},
}
class BudgetExhausted(RuntimeError):
"""Raised rather than silently answering: a caller that has burned its
consult budget is circling, and should escalate or ask the operator."""
@dataclass(frozen=True)
class ExpertAnswer:
answer: str
confidence: str
caveats: str
def render(self) -> str:
body = f"{self.answer}\n\n(confidence: {self.confidence})"
return f"{body}\n\nCaveats: {self.caveats}" if self.caveats else body
class ExpertDesk:
def __init__(
self,
client: LLMClient,
store: StateStore,
roles: RoleMap,
*,
budget: int = DEFAULT_BUDGET,
) -> None:
self._client = client
self._store = store
self._roles = roles
self._budget = budget
def consults_used(self, since_event_id: int = 0) -> int:
return sum(
1
for e in self._store.read_events(after=since_event_id)
if isinstance(e.event, DelegationStarted)
and e.event.role == CONSULTANT_ROLE
)
def ask(self, question: str, *, since_event_id: int = 0) -> ExpertAnswer:
used = self.consults_used(since_event_id)
if used >= self._budget:
raise BudgetExhausted(
f"consult budget of {self._budget} is spent for this task "
f"({used} used); escalate or ask the operator instead"
)
model = self._roles.resolve(CONSULTANT_ROLE)
self._store.append_event(
DelegationStarted(
brief=question,
role=CONSULTANT_ROLE,
model=model,
caps={"consults": self._budget, "used": used},
)
)
answer, note = self._answer(question, model)
self._store.append_event(
DelegationCompleted(
handback=answer.render() if answer else note,
artifacts=[],
tokens=0,
duration_s=0.0,
exit_status="answered" if answer else "failed",
)
)
if answer is None:
raise LLMError(f"ask_expert produced no usable answer: {note}")
return answer
def _answer(self, question: str, model: str) -> tuple[ExpertAnswer | None, str]:
system = SYSTEM_PROMPT
last = "no reply"
for attempt in (1, 2):
try:
# Stateless by design (A14): the question travels as a
# synthesised message that is never appended to the log, so a
# consult neither depends on nor leaks repo context. It cannot
# ride in the system prompt alone — the Messages API requires
# at least one non-system message.
produced = self._client.complete(
[
StoredEvent(
id=0,
ts="",
event=UserMessage(text=question, source="ask_expert"),
)
],
tools=None,
model=model,
system=system,
)
except LLMError as exc:
return None, str(exc)
text = "\n".join(
getattr(e, "text", "") for e in produced if hasattr(e, "text")
).strip()
parsed = parse_answer(text)
if parsed is not None:
return parsed, "ok"
last = text[:300] or "empty reply"
if attempt == 1:
system = SYSTEM_PROMPT + _RETRY_NUDGE
return None, f"unparseable reply: {last}"
def parse_answer(text: str) -> ExpertAnswer | None:
"""Validated locally because the prompt-level contract is all we have.
Tolerates a fenced code block, since models wrap JSON in one habitually,
but not missing or malformed fields — those are a failed turn.
"""
candidate = text.strip()
fenced = re.search(r"```(?:json)?\s*(.*?)```", candidate, re.DOTALL)
if fenced:
candidate = fenced.group(1).strip()
start, end = candidate.find("{"), candidate.rfind("}")
if start == -1 or end <= start:
return None
try:
data = json.loads(candidate[start : end + 1])
except json.JSONDecodeError:
return None
if not isinstance(data, dict):
return None
answer = data.get("answer")
confidence = data.get("confidence")
if not isinstance(answer, str) or not answer.strip():
return None
if not isinstance(confidence, str) or confidence.lower() not in _CONFIDENCES:
return None
caveats = data.get("caveats")
return ExpertAnswer(
answer=answer.strip(),
confidence=confidence.lower(),
caveats=caveats.strip() if isinstance(caveats, str) else "",
)
nisse-dev/src/nisse_dev/git.py
+14
View File
@@ -59,6 +59,10 @@ class GitRepo:
self.root.mkdir(parents=True, exist_ok=True)
self._run("init", "-q", "-b", default_branch)
def clone(self, remote_url: str, dest: Path, *, token: str | None = None) -> None:
self.root.mkdir(parents=True, exist_ok=True)
self._run("clone", "-q", remote_url, str(dest), secret=token)
def configure_identity(self, name: str, email: str) -> None:
self._run("config", "user.name", name)
self._run("config", "user.email", email)
@@ -66,6 +70,16 @@ class GitRepo:
def checkout_new_branch(self, name: str) -> None:
self._run("checkout", "-q", "-b", name)
def checkout_existing_branch(self, name: str) -> None:
self._run("checkout", "-q", name)
def branch_exists(self, name: str) -> bool:
try:
self._run("rev-parse", "--verify", "--quiet", f"refs/heads/{name}")
except GitError:
return False
return True
def commit_all(self, message: str) -> str | None:
"""Returns the new commit sha, or None when there was nothing to
commit — an empty milestone is a fact to report, not an error."""
nisse-dev/src/nisse_dev/gitea.py
-10
View File
@@ -10,8 +10,6 @@ idempotent across a crash: boot reconciliation (A3) asks Gitea whether the
effect already happened rather than repeating it.
"""
import os
from collections.abc import Mapping
from dataclasses import dataclass
from typing import Any
@@ -60,14 +58,6 @@ class GiteaClient:
transport=transport,
)
@classmethod
def from_env(cls, env: Mapping[str, str] = os.environ) -> "GiteaClient":
base_url = env.get("NISSE_GITEA_URL")
token = env.get("NISSE_GITEA_TOKEN")
if not base_url or not token:
raise GiteaError("set NISSE_GITEA_URL and NISSE_GITEA_TOKEN")
return cls(base_url, token)
def _request(self, method: str, path: str, **kwargs: Any) -> Any:
try:
response = self._http.request(method, path, **kwargs)
nisse-dev/src/nisse_dev/llm.py
+167 -46
View File
@@ -2,18 +2,16 @@
One client interface, present from the first real LLM call. All traffic goes
to the LiteLLM proxy — the VM's single LLM destination; the agent holds one
LiteLLM token and no vendor credentials. Config comes from the environment
(``NISSE_LLM_BASE_URL``, ``NISSE_LLM_TOKEN``), nothing hardcoded.
LiteLLM token and no vendor credentials. The base URL and token come from the
agent's config (nisse_dev.config), nothing hardcoded.
Model identity is expressed as roles (A11), resolved to concrete model strings
from env config; the resolved mapping is pinned at task start so a mid-task
from config; the resolved mapping is pinned at task start so a mid-task
config change cannot split a task across two planners.
"""
import json
import os
import time
from collections.abc import Mapping
from dataclasses import asdict, dataclass
from typing import Any, ClassVar, Protocol
@@ -41,6 +39,55 @@ class LLMError(RuntimeError):
RATE_LIMIT_STATUS = 429
RATE_LIMIT_RETRY_S = 20.0
# Generous by default. Output is billed by what is produced, not by the
# ceiling, and a ceiling too low does not fail loudly — it truncates a tool
# call mid-argument and hands back something that parses. Measured: Opus
# writing an architecture plan routinely spends 3,000-4,000 output tokens and
# was observed at 4,064 against a 4,096 limit.
DEFAULT_MAX_OUTPUT_TOKENS = 16384
class UpstreamUnavailable(LLMError):
"""The proxy or provider could not be reached, or refused for reasons
unrelated to what was asked: a network blip, a timeout, a 5xx, a rate
limit that did not clear, an exhausted budget.
Worth distinguishing because the *work* is not at fault. A delegation
stopped by one of these has a branch that may be perfectly good, so it is
preserved and resumed rather than discarded — unlike a delegate that made
a mess, which is cleaned up.
"""
class BudgetExhausted(UpstreamUnavailable):
"""The proxy refused because a spend limit is gone.
Distinct from the rest of its family because it will not clear on its
own within the hour: an exhausted daily budget takes external models away
until it resets, so retrying wastes time and the operator needs telling.
A5 keeps the budget itself at the proxy; recognising its refusal is ours.
"""
def _looks_like_budget_refusal(text: str) -> bool:
# Pattern-matched rather than contracted: LiteLLM's wording is not a
# documented interface, so this errs toward missing one rather than
# misreporting an unrelated failure as a budget problem.
lowered = text.lower()
return "budget" in lowered and (
"exceed" in lowered or "exhaust" in lowered or "limit" in lowered
)
class TruncatedResponse(LLMError):
"""The model was cut off mid-answer.
Raised rather than returned, because the damage is silent: a truncated
tool call arrives with required arguments simply absent, and a truncated
write_file would write half a file and report success. A cut-off turn is
a failed turn, and every caller already retries or gates on LLMError.
"""
class LLMClient(Protocol):
def complete(
@@ -59,22 +106,10 @@ class RoleMap:
delegate: str
consultant: str
_ENV_VARS: ClassVar[dict[str, str]] = {
"executor": "NISSE_ROLE_EXECUTOR",
"planner": "NISSE_ROLE_PLANNER",
"delegate": "NISSE_ROLE_DELEGATE",
"consultant": "NISSE_ROLE_CONSULTANT",
}
@classmethod
def from_env(cls, env: Mapping[str, str] = os.environ) -> "RoleMap":
missing = [var for var in cls._ENV_VARS.values() if not env.get(var)]
if missing:
raise LLMError(f"role mapping incomplete; set: {', '.join(missing)}")
return cls(**{role: env[var] for role, var in cls._ENV_VARS.items()})
ROLES: ClassVar[tuple[str, ...]] = ("executor", "planner", "delegate", "consultant")
def resolve(self, role: str) -> str:
if role not in self._ENV_VARS:
if role not in self.ROLES:
raise LLMError(f"unknown role: {role!r}")
return str(getattr(self, role))
@@ -92,6 +127,7 @@ class LiteLLMClient:
timeout_s: float = 120.0,
transport: httpx.BaseTransport | None = None,
sleep: Any = time.sleep,
max_tokens: int | None = None,
) -> None:
self._http = httpx.Client(
base_url=base_url.rstrip("/"),
@@ -101,14 +137,16 @@ class LiteLLMClient:
)
self._renderer = OpenAIChatRenderer()
self._sleep = sleep
@classmethod
def from_env(cls, env: Mapping[str, str] = os.environ) -> "LiteLLMClient":
base_url = env.get("NISSE_LLM_BASE_URL")
token = env.get("NISSE_LLM_TOKEN")
if not base_url or not token:
raise LLMError("set NISSE_LLM_BASE_URL and NISSE_LLM_TOKEN")
return cls(base_url, token)
self._max_tokens = max_tokens or DEFAULT_MAX_OUTPUT_TOKENS
# Set after each call so a caller can meter itself (A13 caps);
# a side channel rather than a return value, so the LLMClient
# protocol stays the same for every implementation.
self.last_usage: dict[str, int] = {}
self.last_model: str | None = None
self.last_cost: float = 0.0
# Set even when the call then fails: time spent waiting out a rate
# limit is the proxy's, not the model's (the benchmark subtracts it).
self.last_rate_limit_wait_s: float = 0.0
def complete(
self,
@@ -116,25 +154,54 @@ class LiteLLMClient:
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
*,
max_tokens: int | None = None,
require_tool_call: bool = False,
) -> list[CanonicalEvent]:
body = self._renderer.render(events, tools, model, system=system)
response = _post_with_rate_limit_retry(
if require_tool_call and tools:
# Only for models whose server enforces it; see BudgetedClient.
body["tool_choice"] = "required"
# A ceiling on this route too: without one a runaway turn is bounded
# only by the provider's default, and the same truncation check has
# nothing to check against. A caller that knows the model's limits
# (the context budget) passes a tighter one.
body["max_tokens"] = max_tokens or self._max_tokens
self.last_rate_limit_wait_s = 0.0 # not the last call's, if this one raises
response, self.last_rate_limit_wait_s = _post_with_rate_limit_retry(
self._http, "/chat/completions", body, self._sleep
)
if response.status_code != 200:
_raise_for_upstream(response)
raise LLMError(
f"LLM request returned {response.status_code}: {response.text[:500]}"
)
# LiteLLM prices every call; kept so a caller can meter spending.
self.last_cost = _response_cost(response)
return self._parse(response.json(), requested_model=model)
def _parse(
self, data: dict[str, Any], requested_model: str
) -> list[CanonicalEvent]:
try:
message = data["choices"][0]["message"]
choice = data["choices"][0]
message = choice["message"]
except (KeyError, IndexError) as exc:
raise LLMError(f"malformed completion response: {exc}") from exc
if choice.get("finish_reason") == "length":
raise TruncatedResponse(
f"{requested_model} hit its output limit mid-answer; raise "
"llm.max_output_tokens, or the model's max_output_tokens in LiteLLM"
)
produced_by = data.get("model") or requested_model
# What actually answered, which is not always what was asked for: a
# proxy fallback answers under another model's name.
self.last_model = produced_by
usage = data.get("usage") or {}
self.last_usage = {
"input": int(usage.get("prompt_tokens") or 0),
"output": int(usage.get("completion_tokens") or 0),
}
out: list[CanonicalEvent] = []
if message.get("content"):
out.append(AssistantText(text=message["content"], model=produced_by))
@@ -184,7 +251,7 @@ class AnthropicPassthroughClient:
token: str,
*,
timeout_s: float = 300.0,
max_tokens: int = 4096,
max_tokens: int | None = None,
transport: httpx.BaseTransport | None = None,
sleep: Any = time.sleep,
) -> None:
@@ -199,18 +266,14 @@ class AnthropicPassthroughClient:
transport=transport,
)
self._renderer = AnthropicMessagesRenderer()
self._max_tokens = max_tokens
self._max_tokens = max_tokens or DEFAULT_MAX_OUTPUT_TOKENS
self._sleep = sleep
@classmethod
def from_env(
cls, env: Mapping[str, str] = os.environ
) -> "AnthropicPassthroughClient":
base_url = env.get("NISSE_LLM_BASE_URL")
token = env.get("NISSE_LLM_TOKEN")
if not base_url or not token:
raise LLMError("set NISSE_LLM_BASE_URL and NISSE_LLM_TOKEN")
return cls(base_url, token)
self.last_usage: dict[str, int] = {}
self.last_model: str | None = None
self.last_cost: float = 0.0
# Set even when the call then fails: time spent waiting out a rate
# limit is the proxy's, not the model's (the benchmark subtracts it).
self.last_rate_limit_wait_s: float = 0.0
def complete(
self,
@@ -218,26 +281,52 @@ class AnthropicPassthroughClient:
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
*,
max_tokens: int | None = None,
require_tool_call: bool = False,
) -> list[CanonicalEvent]:
body = self._renderer.render(
events, tools, model, system=system, max_tokens=self._max_tokens
events,
tools,
model,
system=system,
max_tokens=max_tokens or self._max_tokens,
)
response = _post_with_rate_limit_retry(
if require_tool_call and tools:
body["tool_choice"] = {"type": "any"} # Anthropic's name for it
self.last_rate_limit_wait_s = 0.0 # not the last call's, if this one raises
response, self.last_rate_limit_wait_s = _post_with_rate_limit_retry(
self._http, "/v1/messages", body, self._sleep
)
if response.status_code != 200:
_raise_for_upstream(response)
raise LLMError(
f"passthrough returned {response.status_code}: {response.text[:500]}"
)
# LiteLLM prices every call; kept so a caller can meter spending.
self.last_cost = _response_cost(response)
return self._parse(response.json(), requested_model=model)
def _parse(
self, data: dict[str, Any], requested_model: str
) -> list[CanonicalEvent]:
if data.get("stop_reason") == "max_tokens":
raise TruncatedResponse(
f"{requested_model} hit its output limit mid-answer; raise "
"llm.max_output_tokens, or the model's max_output_tokens in LiteLLM"
)
content = data.get("content")
if not isinstance(content, list):
raise LLMError(f"malformed passthrough response: {str(data)[:300]}")
produced_by = data.get("model") or requested_model
# What actually answered, which is not always what was asked for: a
# proxy fallback answers under another model's name.
self.last_model = produced_by
usage = data.get("usage") or {}
self.last_usage = {
"input": int(usage.get("input_tokens") or 0),
"output": int(usage.get("output_tokens") or 0),
}
out: list[CanonicalEvent] = []
for block in content:
kind = block.get("type")
@@ -266,13 +355,45 @@ class AnthropicPassthroughClient:
def _post_with_rate_limit_retry(
http: httpx.Client, path: str, body: dict[str, Any], sleep: Any
) -> httpx.Response:
) -> tuple[httpx.Response, float]:
"""The response, and how long was spent waiting out a rate limit."""
waited = 0.0
for attempt in (1, 2):
try:
response = http.post(path, json=body)
except httpx.HTTPError as exc:
raise LLMError(f"LLM request failed: {exc}") from exc
# Could not reach the proxy at all: nothing about the request was
# wrong, so callers holding work in progress should keep it.
raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc
if response.status_code != RATE_LIMIT_STATUS or attempt == 2:
return response
return response, waited
sleep(RATE_LIMIT_RETRY_S)
waited += RATE_LIMIT_RETRY_S
raise AssertionError("unreachable")
# Model families we have a native renderer adapter for. A13: delegations take
# the vendor's passthrough route because a normalising middlebox is lossy
# exactly where a long-running delegation lives.
def _response_cost(response: httpx.Response) -> float:
try:
return float(response.headers.get("x-litellm-response-cost") or 0.0)
except ValueError:
return 0.0
def _raise_for_upstream(response: httpx.Response) -> None:
"""Classify a non-200 as upstream trouble where it plainly is.
A 4xx that is not a rate limit means the request itself was wrong, and
preserving work on the strength of one would hide a real bug — so the
benefit of the doubt stops at the server's own door.
"""
if _looks_like_budget_refusal(response.text):
raise BudgetExhausted(
f"the proxy refused on budget grounds: {response.text[:300]}"
)
if response.status_code >= 500 or response.status_code == RATE_LIMIT_STATUS:
raise UpstreamUnavailable(
f"upstream returned {response.status_code}: {response.text[:300]}"
)
nisse-dev/src/nisse_dev/main.py
+40 -5
View File
@@ -1,11 +1,46 @@
"""Placeholder entry point for the dev agent."""
"""Entry point for the dev agent.
from nisse_core.greeting import greeting
Loads configuration, builds the agent, and stays up until told to stop.
Shutdown is deliberate rather than abrupt: A2's durability guarantees assume
the process may die at any moment, but a clean stop still beats relying on
them.
"""
import os
import signal
import sys
from pathlib import Path
from types import FrameType
from nisse_dev.agent import build_agent
from nisse_dev.config import AgentConfig, ConfigError
from nisse_dev.envfile import load_env_file
def main() -> None:
print(greeting("nisse-dev"))
def main() -> int:
load_env_file(Path(os.environ.get("NISSE_ENV", ".env")).expanduser())
try:
config = AgentConfig.load()
except ConfigError as exc:
# Fail here rather than twenty minutes into a milestone on the one
# call that needed the missing value.
print(f"nisse-dev: {exc}", file=sys.stderr)
return 2
agent = build_agent(config)
def shutdown(_signum: int, _frame: FrameType | None) -> None:
print("nisse-dev: stopping", file=sys.stderr)
agent.stop()
signal.signal(signal.SIGINT, shutdown)
signal.signal(signal.SIGTERM, shutdown)
agent.start()
print(f"nisse-dev: ready as {agent.account} on {config.project}", file=sys.stderr)
agent.wait()
return 0
if __name__ == "__main__":
main()
raise SystemExit(main())
nisse-dev/src/nisse_dev/matrix.py
+1 -26
View File
@@ -11,12 +11,10 @@ loop backs off (capped) and retries indefinitely — an outage delays reactions
but never fails anything.
"""
import os
import threading
import uuid
from collections.abc import Callable, Mapping
from collections.abc import Callable
from dataclasses import dataclass
from typing import ClassVar
import httpx
@@ -34,29 +32,6 @@ class MatrixConfig:
token: str
room_id: str
_ENV: ClassVar[tuple[str, ...]] = (
"NISSE_MATRIX_HOMESERVER",
"NISSE_MATRIX_USER_ID",
"NISSE_MATRIX_TOKEN",
)
@classmethod
def from_env(
cls,
env: Mapping[str, str] = os.environ,
room_var: str = "NISSE_MATRIX_ROOM_ID",
) -> "MatrixConfig":
needed = (*cls._ENV, room_var)
missing = [var for var in needed if not env.get(var)]
if missing:
raise TransportError(f"matrix config incomplete; set: {', '.join(missing)}")
return cls(
homeserver=env["NISSE_MATRIX_HOMESERVER"].rstrip("/"),
user_id=env["NISSE_MATRIX_USER_ID"],
token=env["NISSE_MATRIX_TOKEN"],
room_id=env[room_var],
)
class MatrixTransport:
def __init__(
nisse-dev/src/nisse_dev/planner.py
+18 -2
View File
@@ -11,9 +11,10 @@ crash mid-planning leaves a truthful transcript.
from typing import Any
from nisse_dev.cancel import Cancel
from nisse_dev.canonical import AssistantText, ToolCall, ToolResult
from nisse_dev.control import AskHuman, PlanningError, SubmitPlan
from nisse_dev.llm import LLMClient, LLMError, RoleMap
from nisse_dev.control import AskHuman, PlanningError, PlanningStopped, SubmitPlan
from nisse_dev.llm import BudgetExhausted, LLMClient, LLMError, RoleMap
from nisse_dev.state import StateStore
# Plans may be executed by a materially weaker model, which cannot
@@ -84,11 +85,15 @@ class LLMPlanningAgent:
roles: RoleMap,
*,
system_prompt: str = SYSTEM_PROMPT,
cancel: Cancel | None = None,
) -> None:
self._client = client
self._store = store
self._roles = roles
self._system_prompt = system_prompt
# Checked before each turn only. A turn is one model call, and a turn
# that completes ends at a gate or a question, which is a halt anyway.
self._cancel = cancel or Cancel()
def plan(
self,
@@ -105,9 +110,11 @@ class LLMPlanningAgent:
f"{self._system_prompt}\n\n"
f"Current task: draft the {gate} plan for project {project!r}."
)
self._check_stop()
action, first_error = self._try_turn(model, instruction)
if action is not None:
return action
self._check_stop()
action, second_error = self._try_turn(model, f"{instruction}\n\n{_NUDGE}")
if action is None:
detail = second_error or first_error
@@ -117,6 +124,10 @@ class LLMPlanningAgent:
)
return action
def _check_stop(self) -> None:
if self._cancel.requested:
raise PlanningStopped("stopped by the operator")
def _try_turn(
self, model: str, system: str
) -> tuple[SubmitPlan | AskHuman | None, str | None]:
@@ -125,6 +136,11 @@ class LLMPlanningAgent:
output is observable, never the model's own claim)."""
try:
return self._turn(model, system), None
except BudgetExhausted:
# Retrying cannot help: the budget will not reappear within a
# turn. Surface it now so the operator is told rather than the
# loop spending a second call to fail identically.
raise
except LLMError as exc:
return None, str(exc)
nisse-dev/src/nisse_dev/probe.py
+142
View File
@@ -0,0 +1,142 @@
"""Check a model's declared context window against what its server does.
LiteLLM's ``max_input_tokens`` for a local model is only a declaration; the
real limit is whatever the model server allocates. This sends one prompt
half again as long as the declared window and reads back what the server did
with it:
- **It refused.** Over-long prompts fail loudly, which is the safe behaviour.
- **It cut the prompt.** The reported prompt size is where it cut, which is
the server's real window. Ollama does this, silently.
- **It took the whole prompt.** The server allows more than is declared, so
the declaration is conservative.
Every answer is checked for which model gave it. A proxy that falls back to
another model on an error returns a normal-looking answer under that model's
name, and once made this probe report a refusing server as roomy.
Run it with ``scripts/probe_context.py``. It spends one small and one large
request on the model, so it refuses windows above a ceiling unless told
otherwise: commercial models' limits are already known to LiteLLM.
"""
from dataclasses import dataclass
import httpx
from nisse_dev.catalog import ModelFacts
# Of a probe's size, how much must come back for it to count as uncut. A
# chat template adds a little, and a tokenizer counts a little differently.
_UNCUT = 0.9
# How close the server's cut must be to the declaration to count as a match.
_MATCH = 0.02
@dataclass(frozen=True)
class ProbeResult:
verdict: str # "matches", "declared_too_high", "declared_conservative", ...
message: str
server_window: int | None = None
def interpret(
declared: int, sent: int, reported: int | None, error: str | None
) -> ProbeResult:
"""What one over-long probe says about the server's window.
``sent`` is roughly how many tokens the probe held; ``reported`` is the
prompt size the server reported, if it answered; ``error`` its error
text, if it refused.
"""
if error is not None:
return ProbeResult(
"refuses",
f"the server refused a ~{sent}-token prompt ({error[:200]}). Over-long "
"prompts fail loudly here, which is safe; the declared window of "
f"{declared} was not contradicted.",
)
assert reported is not None
if reported >= sent * _UNCUT:
return ProbeResult(
"declared_conservative",
f"the server took the whole ~{sent}-token prompt ({reported} reported), "
f"more than the declared {declared}. The declaration is safe; raise it "
"if you want the room.",
server_window=None,
)
if abs(reported - declared) <= declared * _MATCH:
return ProbeResult(
"matches",
f"the server cut the prompt at {reported} tokens, matching the declared "
f"{declared}.",
server_window=reported,
)
if reported < declared:
return ProbeResult(
"declared_too_high",
f"the server cut the prompt at {reported} tokens, but LiteLLM declares "
f"{declared}. Prompts between the two would be silently cut: set "
f"max_input_tokens to {reported}.",
server_window=reported,
)
return ProbeResult(
"declared_conservative",
f"the server cut the prompt at {reported} tokens, above the declared "
f"{declared}. The declaration is safe; it could be raised to {reported}.",
server_window=reported,
)
def _prompt(words: int) -> str:
return " ".join(f"item{i}" for i in range(words)) + "\n\nReply with just: ok"
class Fallback(Exception):
def __init__(self, answered_by: str) -> None:
self.answered_by = answered_by
def _send(
http: httpx.Client, model: str, words: int, backing: str
) -> tuple[int | None, str | None]:
response = http.post(
"/chat/completions",
json={
"model": model,
"messages": [{"role": "user", "content": _prompt(words)}],
"max_tokens": 5,
},
)
if response.status_code != 200:
return None, f"{response.status_code}: {response.text}"
data = response.json()
answered = str(data.get("model") or "")
facts = ModelFacts(alias=model, backing=backing, provider="", supports_tools=None)
if not facts.answered(answered):
raise Fallback(answered)
usage = data.get("usage") or {}
return int(usage.get("prompt_tokens") or 0), None
def probe(
http: httpx.Client, model: str, declared: int, backing: str = ""
) -> ProbeResult:
"""Measure how many tokens a word of the probe costs on this model, then
send a prompt half again as long as the declared window."""
calibration_words = 200
try:
tokens, error = _send(http, model, calibration_words, backing)
if error is not None or not tokens:
return ProbeResult("failed", f"a short calibration request failed: {error}")
per_word = tokens / calibration_words
sent = int(declared * 1.5)
reported, error = _send(http, model, int(sent / per_word), backing)
except Fallback as fallback:
return ProbeResult(
"fallback",
f"{fallback.answered_by} answered instead of {model}: the proxy fell "
"back to another model, so nothing here describes this one. Remove "
"the fallback, or probe the server directly.",
)
return interpret(declared, sent, reported, error)
nisse-dev/src/nisse_dev/state.py
+43 -2
View File
@@ -27,7 +27,7 @@ from nisse_dev.canonical import (
)
from nisse_dev.oplog import OpLog
CURRENT_SCHEMA_VERSION = 2
CURRENT_SCHEMA_VERSION = 3
Migration = Callable[[sqlite3.Connection], None]
@@ -41,12 +41,29 @@ CREATE TABLE canonical_events (
"""
_EVAL_OUTCOMES_DDL = """
CREATE TABLE eval_outcomes (
event_id INTEGER PRIMARY KEY,
verdict TEXT NOT NULL,
note TEXT NOT NULL DEFAULT '',
marked_at TEXT NOT NULL
)
"""
def _migrate_v1_add_canonical_events(conn: sqlite3.Connection) -> None:
conn.execute(_CANONICAL_EVENTS_DDL)
def _migrate_v2_add_eval_outcomes(conn: sqlite3.Connection) -> None:
conn.execute(_EVAL_OUTCOMES_DDL)
# Maps version N to the migration that takes the schema from N to N+1.
MIGRATIONS: Mapping[int, Migration] = {1: _migrate_v1_add_canonical_events}
MIGRATIONS: Mapping[int, Migration] = {
1: _migrate_v1_add_canonical_events,
2: _migrate_v2_add_eval_outcomes,
}
class StateSchemaError(RuntimeError):
@@ -93,6 +110,7 @@ _SCHEMA = [
)
""",
_CANONICAL_EVENTS_DDL,
_EVAL_OUTCOMES_DDL,
]
@@ -118,6 +136,7 @@ class StateStore:
self._conn.execute("PRAGMA synchronous=FULL")
self._conn.execute("PRAGMA busy_timeout=5000")
self._target_version = schema_version
self.closed = False
self._init_or_migrate(migrations)
def _init_or_migrate(self, migrations: Mapping[int, Migration]) -> None:
@@ -294,8 +313,30 @@ class StateStore:
for row in rows
]
def mark_outcome(self, event_id: int, verdict: str, note: str = "") -> None:
"""Record whether a delegation or consult was good (A16).
Kept out of the canonical log on purpose: the log is an append-only
record of what happened, and a verdict is a later judgement about it
that may be revised.
"""
self._conn.execute(
"INSERT INTO eval_outcomes (event_id, verdict, note, marked_at) "
"VALUES (?, ?, ?, ?) ON CONFLICT(event_id) DO UPDATE SET "
"verdict = excluded.verdict, note = excluded.note, "
"marked_at = excluded.marked_at",
(event_id, verdict, note, _now()),
)
def outcomes(self) -> dict[int, tuple[str, str]]:
rows = self._conn.execute(
"SELECT event_id, verdict, note FROM eval_outcomes"
).fetchall()
return {r["event_id"]: (r["verdict"], r["note"]) for r in rows}
def close(self) -> None:
self._conn.close()
self.closed = True
def __enter__(self) -> Self:
return self
nisse-dev/src/nisse_dev/tools.py
+18 -11
View File
@@ -20,6 +20,8 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any
from nisse_dev.context import head_and_tail
# Tools that change or verify something. Reading a file or listing a
# directory is a model looking around, not doing the work, so neither counts
# as evidence that work was carried out.
@@ -86,13 +88,6 @@ EXECUTOR_TOOLS: list[dict[str, Any]] = [
]
def _truncate(text: str) -> str:
if len(text) <= MAX_OUTPUT_CHARS:
return text
dropped = len(text) - MAX_OUTPUT_CHARS
return f"{text[:MAX_OUTPUT_CHARS]}\n[... {dropped} characters truncated]"
def _child_env() -> dict[str, str]:
"""The project's build and test code is arbitrary; it has no business
reading the agent's LiteLLM, Matrix, or Gitea credentials."""
@@ -106,10 +101,20 @@ class ToolOutcome:
class Workspace:
def __init__(self, root: Path, *, timeout_s: int = DEFAULT_TIMEOUT_S) -> None:
def __init__(
self,
root: Path,
*,
timeout_s: int = DEFAULT_TIMEOUT_S,
max_output_chars: int = MAX_OUTPUT_CHARS,
) -> None:
self.root = Path(root)
self.root.mkdir(parents=True, exist_ok=True)
self._timeout_s = timeout_s
self._max_output_chars = max_output_chars
def _truncate(self, text: str) -> str:
return head_and_tail(text, self._max_output_chars)
def _resolve(self, path: str) -> Path:
candidate = Path(path)
@@ -142,7 +147,7 @@ class Workspace:
def read_file(self, path: str) -> ToolOutcome:
target = self._resolve(path)
try:
return ToolOutcome(_truncate(target.read_text()), success=True)
return ToolOutcome(self._truncate(target.read_text()), success=True)
except (OSError, UnicodeDecodeError) as exc:
return ToolOutcome(f"could not read {path}: {exc}", success=False)
@@ -175,7 +180,9 @@ class Workspace:
)
except OSError as exc:
return ToolOutcome(f"could not list {path}: {exc}", success=False)
return ToolOutcome(_truncate("\n".join(entries) or "(empty)"), success=True)
return ToolOutcome(
self._truncate("\n".join(entries) or "(empty)"), success=True
)
def describe(self) -> str:
"""Told to the model, so it does not have to guess where it is."""
@@ -212,7 +219,7 @@ class Workspace:
f"command timed out after {self._timeout_s}s: {command}",
success=False,
)
body = _truncate(
body = self._truncate(
f"exit code: {process.returncode}\nstdout:\n{stdout}\nstderr:\n{stderr}"
)
return ToolOutcome(body, success=process.returncode == 0)
nisse-dev/src/nisse_dev/worker.py
+138
View File
@@ -0,0 +1,138 @@
"""The one thread that runs orchestrator work and owns the state store.
Planning turns, milestones and review responses can each run for minutes.
Running them on the thread that delivered the message blocked that thread for
as long: the Matrix sync loop read nothing new until the work finished, so an
operator's `stop` (A4) would only have arrived after the work it was meant to
stop. Every handler now runs here instead, and transports only hand work over.
Owning the store is the other half. SQLite connections are bound to the thread
that opened them, and the store keeps that check on: it is opened on this
thread, used only from it, and any future code that touches it from elsewhere
fails loudly instead of racing.
Jobs run one at a time, in order, so state changes never interleave — the
guarantee the orchestrator's lock used to give.
"""
import sys
import threading
import traceback
from collections import deque
from collections.abc import Callable
from concurrent.futures import Future
from queue import Queue
from typing import Protocol, TypeVar
T = TypeVar("T")
class Worker(Protocol):
@property
def busy(self) -> bool: ...
def submit(self, job: Callable[[], object]) -> None: ...
def call(self, job: Callable[[], T], timeout: float | None = None) -> T: ...
class InlineWorker:
"""Runs jobs on the caller's thread, for tests and single-threaded use.
A job submitted from inside a running job waits until that one finishes,
exactly as it would queue behind it on the real worker. Without that, a
test that says `stop` from inside a fake milestone would run the stop
handling in the middle of the milestone, which production never does.
"""
def __init__(self) -> None:
self._pending: deque[Callable[[], object]] = deque()
self._running = False
@property
def busy(self) -> bool:
return self._running
def submit(self, job: Callable[[], object]) -> None:
self._pending.append(job)
if self._running:
return
self._running = True
try:
while self._pending:
self._pending.popleft()()
finally:
self._running = False
def call(self, job: Callable[[], T], timeout: float | None = None) -> T:
if self._running:
return job() # already on the worker, as a nested call would be
box: list[T] = []
self.submit(lambda: box.append(job()))
return box[0]
_STOP = object()
def _report(exc: BaseException) -> None:
traceback.print_exception(exc, file=sys.stderr)
class SerialWorker:
def __init__(
self,
*,
on_error: Callable[[BaseException], None] = _report,
name: str = "nisse-worker",
) -> None:
self._queue: Queue[object] = Queue()
self._on_error = on_error
self._busy = False
self._thread = threading.Thread(target=self._run, name=name, daemon=True)
@property
def busy(self) -> bool:
return self._busy
def start(self) -> None:
self._thread.start()
def submit(self, job: Callable[[], object]) -> None:
self._queue.put((job, None))
def call(self, job: Callable[[], T], timeout: float | None = None) -> T:
"""Run a job here and wait for its result. Raises TimeoutError if it
has not finished in time; the job itself still runs."""
if threading.current_thread() is self._thread:
return job()
future: Future[T] = Future()
self._queue.put((job, future))
return future.result(timeout)
def stop(self, timeout: float | None = None) -> None:
self._queue.put(_STOP)
self._thread.join(timeout)
def _run(self) -> None:
while True:
item = self._queue.get()
if item is _STOP:
return
job, future = item # type: ignore[misc]
self._busy = True
try:
result = job()
except Exception as exc: # noqa: BLE001 -- see below
# One failed job must not take the agent's only worker with
# it. A caller that waits gets the exception; otherwise it is
# reported and the next job runs.
if future is not None:
future.set_exception(exc)
else:
self._on_error(exc)
else:
if future is not None:
future.set_result(result)
finally:
self._busy = False
nisse-dev/tests/test_agent_build.py
+349
View File
@@ -0,0 +1,349 @@
"""The composition root. These tests exist because two components — the
delegation harness and the comment poller — were fully built, fully tested,
and reachable from nothing but their own tests. Unit tests that construct a
component directly cannot catch that; only assembly can."""
from pathlib import Path
from typing import Any
import httpx
import pytest
from test_config import SECRETS, tree
from nisse_dev.agent import build_agent, discover_models, prepare_workspace
from nisse_dev.catalog import ModelCatalog, UnknownModel
from nisse_dev.config import AgentConfig
from nisse_dev.git import GitRepo
from nisse_dev.oplog import OpLog
# What the test roles resolve to, as LiteLLM would report them.
TEST_CATALOG = ModelCatalog.from_payload(
{
"data": [
{
"model_name": alias,
"litellm_params": {"model": f"anthropic/{alias}"},
"model_info": {
"litellm_provider": "anthropic",
"supports_function_calling": True,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
},
}
for alias in ("claude-opus-5", "claude-haiku-4-5")
]
}
)
def config_for(tmp_path: Path, **changes: str) -> AgentConfig:
return AgentConfig.from_tree(
tree(
state_db=str(tmp_path / "state.db"),
workspace=str(tmp_path / "work"),
**changes,
),
SECRETS,
)
@pytest.fixture
def origin(tmp_path: Path) -> Path:
"""A bare repo standing in for the project on the forge."""
remote = tmp_path / "origin.git"
GitRepo(tmp_path)._run("init", "-q", "--bare", "-b", "main", str(remote))
seed = GitRepo(tmp_path / "seed")
seed.init()
seed.configure_identity("seed", "seed@test")
(seed.root / "README.md").write_text("project\n")
seed.commit_all("seed")
seed.push(str(remote), "main")
return remote
def test_workspace_is_cloned_when_absent(tmp_path: Path, origin: Path) -> None:
config = config_for(tmp_path, gitea__url=f"file://{origin}")
# authenticated_url would mangle a file:// remote, so clone directly.
GitRepo(tmp_path).clone(str(origin), config.workspace)
repo = prepare_workspace(config, "nisse-dev-testing")
assert (config.workspace / "README.md").exists()
assert repo.current_branch() == "main"
def test_workspace_is_reused_when_already_cloned(tmp_path: Path, origin: Path) -> None:
config = config_for(tmp_path)
GitRepo(tmp_path).clone(str(origin), config.workspace)
(config.workspace / "local-work.txt").write_text("in progress")
prepare_workspace(config, "nisse-dev-testing")
# Re-cloning would silently destroy uncommitted work.
assert (config.workspace / "local-work.txt").exists()
class FakeTransport:
def __init__(self) -> None:
self.sent: list[str] = []
self.started = False
self.closed = False
def send_message(self, text: str) -> None:
self.sent.append(text)
def send_approval_request(self, text: str) -> None:
self.sent.append(text)
def ask_question(self, text: str) -> None:
self.sent.append(text)
def on_message(self, callback: Any) -> None:
self.callback = callback
def start(self) -> None:
self.started = True
def close(self) -> None:
self.closed = True
def build_offline(tmp_path: Path, origin: Path, monkeypatch: Any) -> Any:
"""Assemble the real graph with only the network boundaries faked."""
config = config_for(tmp_path)
GitRepo(tmp_path).clone(str(origin), config.workspace)
monkeypatch.setattr(
"nisse_dev.agent.GiteaClient.whoami", lambda self: "nisse-dev-testing"
)
monkeypatch.setattr(
"nisse_dev.agent.ModelCatalog.discover",
classmethod(lambda cls, *a, **kw: TEST_CATALOG),
)
monkeypatch.setattr("nisse_dev.agent.MatrixTransport", lambda cfg: FakeTransport())
monkeypatch.setattr(
"nisse_dev.agent.authenticated_url",
lambda base, repo, user, token: str(origin),
)
return build_agent(config)
def test_the_whole_graph_assembles(tmp_path: Path, origin: Path, monkeypatch) -> None:
agent = build_offline(tmp_path, origin, monkeypatch)
orchestrator = agent.orchestrator
# Every component that was previously unreachable is now connected.
assert orchestrator._executor is not None
assert orchestrator._publisher is not None
assert orchestrator._board is not None
assert orchestrator._delegate is not None
assert orchestrator._poller_factory is not None
assert orchestrator._executor._delegate is not None # A12 escalation path
assert orchestrator._executor._expert is not None # A14 consult path
agent.stop()
def test_start_announces_where_it_resumed(
tmp_path: Path, origin: Path, monkeypatch
) -> None:
agent = build_offline(tmp_path, origin, monkeypatch)
agent.start()
assert agent.transport.started
readiness = agent.transport.sent[-1]
assert "nisse-dev up as nisse-dev-testing" in readiness
assert "greet" in readiness # the project
assert "claude-opus-5" in readiness # the pinned planner
agent.stop()
def test_stop_closes_everything_and_releases_the_wait(
tmp_path: Path, origin: Path, monkeypatch
) -> None:
agent = build_offline(tmp_path, origin, monkeypatch)
agent.start()
agent.stop()
assert agent.transport.closed
agent.wait() # returns immediately rather than hanging
assert agent.store.closed
def test_poller_factory_ignores_the_agents_own_comments(
tmp_path: Path, origin: Path, monkeypatch
) -> None:
# Without this the agent answers its own PR replies in a loop.
agent = build_offline(tmp_path, origin, monkeypatch)
poller = agent.orchestrator._poller_factory(lambda c: None)
assert "nisse-dev-testing" in poller._ignore
agent.stop()
def test_delegation_uses_the_passthrough_route(
tmp_path: Path, origin: Path, monkeypatch
) -> None:
# A13: delegations run vendor-native; the normalized route is for the
# places where fidelity is cheap.
from nisse_dev.llm import AnthropicPassthroughClient, LiteLLMClient
agent = build_offline(tmp_path, origin, monkeypatch)
delegation = agent.orchestrator._delegate
assert isinstance(delegation._client._inner, AnthropicPassthroughClient)
assert isinstance(agent.orchestrator._executor._client._inner, LiteLLMClient)
agent.stop()
def test_httpx_clients_are_not_shared_between_routes(
tmp_path: Path, origin: Path, monkeypatch
) -> None:
agent = build_offline(tmp_path, origin, monkeypatch)
normalized = agent.orchestrator._executor._client._inner._http
passthrough = agent.orchestrator._delegate._client._inner._http
assert isinstance(normalized, httpx.Client)
assert normalized is not passthrough # different base URLs and headers
agent.stop()
# --- startup model discovery (A5/A11) ------------------------------------
def test_a_typod_role_refuses_to_boot(tmp_path: Path, monkeypatch) -> None:
# Misconfiguration should cost a second at startup, not be discovered ten
# minutes into a milestone.
config = config_for(tmp_path, roles__planner="clause-opus-5")
monkeypatch.setattr(
"nisse_dev.agent.ModelCatalog.discover",
classmethod(lambda cls, *a, **kw: TEST_CATALOG),
)
with pytest.raises(UnknownModel, match="planner='clause-opus-5'"):
discover_models(config, OpLog("test"))
def test_a_role_this_key_cannot_reach_refuses_to_boot(
tmp_path: Path, monkeypatch
) -> None:
# /model/info is scoped per key, so the same check catches a role pointed
# at a real model that this token has no access to.
config = config_for(tmp_path, roles__executor="local-fast")
monkeypatch.setattr(
"nisse_dev.agent.ModelCatalog.discover",
classmethod(lambda cls, *a, **kw: TEST_CATALOG),
)
with pytest.raises(UnknownModel, match="executor='local-fast'"):
discover_models(config, OpLog("test"))
def test_a_tool_less_model_warns_but_still_boots(tmp_path: Path, monkeypatch) -> None:
# A15 expects weak local models to be accommodated, and models LiteLLM
# declares tool-less are observed driving tool loops anyway.
import io
tool_less = ModelCatalog.from_payload(
{
"data": [
{
"model_name": alias,
"litellm_params": {"model": "openai/qwen3-8b-awq"},
"model_info": {
"litellm_provider": "openai",
"supports_function_calling": False,
"max_input_tokens": 32768,
},
}
for alias in ("claude-opus-5", "claude-haiku-4-5")
]
}
)
monkeypatch.setattr(
"nisse_dev.agent.ModelCatalog.discover",
classmethod(lambda cls, *a, **kw: tool_less),
)
stream = io.StringIO()
discover_models(config_for(tmp_path), OpLog("test", stream))
logged = stream.getvalue()
assert "model_tools_unsupported" in logged
assert '"role": "executor"' in logged
assert '"role": "consultant"' not in logged # ask_expert drives no tools
def test_the_resolved_model_is_pinned_at_task_start(
tmp_path: Path, origin: Path, monkeypatch
) -> None:
agent = build_offline(tmp_path, origin, monkeypatch)
# The store belongs to the worker thread, so the work has to run there.
agent.worker.call(lambda: agent.orchestrator._begin_project("greet"))
pinned = agent.orchestrator._data["models"]
assert pinned["planner"] == {
"alias": "claude-opus-5",
"backing": "anthropic/claude-opus-5",
"provider": "anthropic",
}
agent.stop()
def test_delegation_records_what_the_alias_resolved_to(
tmp_path: Path, origin: Path, monkeypatch
) -> None:
# A16 eval history keyed on an alias alone would average across whatever
# that alias pointed at over time.
agent = build_offline(tmp_path, origin, monkeypatch)
assert agent.orchestrator._delegate._backing_model == "anthropic/claude-opus-5"
agent.stop()
def test_tuning_reaches_every_component(
tmp_path: Path, origin: Path, monkeypatch
) -> None:
from dataclasses import replace
from nisse_dev.config import (
DelegationTuning,
ExecutorTuning,
ExpertTuning,
ToolsTuning,
Tuning,
)
config = replace(
config_for(tmp_path),
max_output_tokens=8192,
tuning=Tuning(
executor=ExecutorTuning(max_turns=40, failure_limit=3),
expert=ExpertTuning(consult_budget=5),
delegation=DelegationTuning(
max_turns=12, max_tokens=50_000, wall_clock_s=60
),
tools=ToolsTuning(command_timeout_s=300, max_output_chars=2000),
),
)
GitRepo(tmp_path).clone(str(origin), config.workspace)
monkeypatch.setattr(
"nisse_dev.agent.GiteaClient.whoami", lambda self: "nisse-dev-testing"
)
monkeypatch.setattr(
"nisse_dev.agent.ModelCatalog.discover",
classmethod(lambda cls, *a, **kw: TEST_CATALOG),
)
monkeypatch.setattr("nisse_dev.agent.MatrixTransport", lambda cfg: FakeTransport())
monkeypatch.setattr(
"nisse_dev.agent.authenticated_url",
lambda base, repo, user, token: str(origin),
)
agent = build_agent(config)
executor = agent.orchestrator._executor
delegation = agent.orchestrator._delegate
assert (executor._max_turns, executor._failure_limit) == (40, 3)
assert executor._expert._budget == 5
assert delegation._caps.turns == 12
assert delegation._caps.tokens == 50_000
assert delegation._caps.wall_clock_s == 60
assert executor._workspace._timeout_s == 300
assert executor._workspace._max_output_chars == 2000
assert executor._client._inner._max_tokens == 8192
assert delegation._client._inner._max_tokens == 8192
assert executor._client._max_output_tokens == 8192
# tool_choice is asked of exactly the models LiteLLM declares enforce it.
assert executor._client._enforces_tool_choice("claude-opus-5") is False
agent.stop()
nisse-dev/tests/test_bench.py
+336
View File
@@ -0,0 +1,336 @@
"""The executor benchmark."""
import json
import shutil
from pathlib import Path
from typing import Any
import pytest
from nisse_dev.bench.results import (
append,
load,
render,
results_file,
summarise,
wilson,
)
from nisse_dev.bench.runner import MeteringClient, run_task
from nisse_dev.bench.tasks import CHECK_DIR, load_tasks
from nisse_dev.canonical import AssistantText, ToolCall
from nisse_dev.catalog import ModelFacts
from nisse_dev.config import ExecutorTuning, Tuning
TASKS = load_tasks()
FACTS = ModelFacts(
alias="local-fast",
backing="openai/qwen3-8b-awq",
provider="openai",
supports_tools=True,
max_input_tokens=32768,
max_output_tokens=8192,
supports_tool_choice=True,
)
# --- the tasks themselves ---------------------------------------------------
def test_the_bundled_tasks_are_the_four_planned() -> None:
assert [t.name for t in TASKS] == [
"fix-bug",
"long-output",
"multi-file",
"write-function",
]
@pytest.mark.parametrize("task", TASKS, ids=lambda t: t.name)
def test_the_seed_fails_its_check_and_the_solution_passes(task, tmp_path: Path) -> None:
# A check that passes on the untouched seed measures nothing; a task its
# own reference solution can't pass is unfair to every model.
seed = tmp_path / "seed"
task.seed_into(seed)
assert not task.run_check(seed)[0], f"{task.name}: the seed already passes"
solved = tmp_path / "solved"
task.seed_into(solved)
task.apply_solution(solved)
passed, output = task.run_check(solved)
assert passed, f"{task.name}: the reference solution fails:\n{output}"
@pytest.mark.parametrize("task", TASKS, ids=lambda t: t.name)
def test_the_hidden_check_is_not_in_the_seed(task, tmp_path: Path) -> None:
task.seed_into(tmp_path)
assert not (tmp_path / CHECK_DIR).exists()
assert "Call finish" in task.plan # plans end the way the executor must
def test_long_output_really_is_long(tmp_path: Path) -> None:
"""The point of the task is output far past tools.max_output_chars."""
import subprocess
import sys
[task] = load_tasks(["long-output"])
task.seed_into(tmp_path)
run = subprocess.run(
[sys.executable, "-m", "pytest", "-v"],
cwd=tmp_path,
capture_output=True,
text=True,
check=False,
)
assert len(run.stdout) > 4 * Tuning().tools.max_output_chars
assert "R237" in run.stdout[-3000:] # the failure is named at the end
def test_a_changed_task_gets_a_new_hash(tmp_path: Path) -> None:
[task] = load_tasks(["write-function"])
shutil.copytree(task.path, tmp_path / "write-function")
[copy] = load_tasks(root=tmp_path)
assert copy.content_hash == task.content_hash
(copy.path / "seed" / "README.md").write_text("changed\n")
assert copy.content_hash != task.content_hash
def test_an_unknown_task_is_refused() -> None:
with pytest.raises(ValueError, match="unknown task"):
load_tasks(["nope"])
# --- running a task ----------------------------------------------------------
class Scripted:
"""A model that plays back fixed turns, reporting itself as `answered`."""
def __init__(self, turns: list[list[Any]], answered: str = "local-fast") -> None:
self.turns = turns
self.last_usage = {"input": 1000, "output": 50}
self.last_cost = 0.0
self.last_model = answered
self.trimmed_requests = 0
def complete(self, events, tools, model, system=None):
if not self.turns:
return [AssistantText(text="nothing left to do", model=model)]
return self.turns.pop(0)
def call(tool: str, cid: str, **arguments: Any) -> ToolCall:
return ToolCall(call_id=cid, tool=tool, arguments=arguments)
SOLVE_SLUGIFY = [
[
call(
"write_file",
"w",
path="slugify.py",
content=(
"import re\n\n\ndef slugify(text):\n"
" return re.sub(r'[^a-z0-9]+', '-', text.lower()).strip('-')\n"
),
)
],
[call("run_command", "r", command="python -c 'import slugify'")],
[call("finish", "f", summary="added slugify")],
]
def test_a_solved_task_is_a_success(tmp_path: Path) -> None:
[task] = load_tasks(["write-function"])
record = run_task(task, FACTS, Scripted(list(SOLVE_SLUGIFY)), Tuning(), note="n")
assert record.success
assert record.outcome == "completed"
assert record.turns == 3
assert record.tokens_in == 3000
assert record.task_hash == task.content_hash
assert record.model_info["supports_tool_choice"] is True
assert record.tuning["executor"]["max_turns"] == 40
def test_saying_finished_is_not_success(tmp_path: Path) -> None:
# Success is the hidden check passing, never the model's own claim.
[task] = load_tasks(["write-function"])
wrong = [
[
call(
"write_file",
"w",
path="slugify.py",
content="def slugify(t):\n return t\n",
)
],
[call("finish", "f", summary="done")],
]
record = run_task(task, FACTS, Scripted(wrong), Tuning())
assert record.outcome == "completed"
assert not record.success
def test_misbehaviour_is_counted(tmp_path: Path) -> None:
[task] = load_tasks(["write-function"])
turns = [
[call("chat", "x1")],
[call("run_command", "r1", command="exit 1")],
[AssistantText(text="let me explain", model="m")],
]
# A higher failure limit, so the invented tool and the failed command
# don't escalate before the prose turn is reached.
tuning = Tuning(executor=ExecutorTuning(failure_limit=5))
record = run_task(task, FACTS, Scripted(turns), tuning)
assert record.invented_tools == 1
assert record.failed_commands == 1
assert record.prose_turns >= 1
assert not record.success
def test_a_fallback_answer_can_never_be_a_success(tmp_path: Path) -> None:
[task] = load_tasks(["write-function"])
client = Scripted(list(SOLVE_SLUGIFY), answered="claude-sonnet-5")
record = run_task(task, FACTS, client, Tuning())
assert record.fallbacks == ["claude-sonnet-5"] * 3
assert not record.success # the check passed, but not for this model
# --- storing and reporting ---------------------------------------------------
def test_wilson_intervals_behave() -> None:
low, high = wilson(5, 5)
assert high == 1.0 and 0.5 < low < 0.6
# 3 of 5 and 5 of 5 overlap: five runs do not separate them.
assert wilson(3, 5)[1] > wilson(5, 5)[0]
assert wilson(0, 0) == (0.0, 1.0)
def record_json(**overrides: Any) -> dict[str, Any]:
[task] = load_tasks(["write-function"])
base = run_task(task, FACTS, Scripted(list(SOLVE_SLUGIFY)), Tuning()).to_json()
base.update(overrides)
return base
def test_results_round_trip_and_report(tmp_path: Path) -> None:
from nisse_dev.bench.runner import RunRecord
rec = RunRecord(**record_json())
path = results_file(tmp_path, "local-fast")
append(path, rec)
append(path, RunRecord(**record_json(success=False)))
stored = load(tmp_path)
assert len(stored) == 2
assert json.loads(path.read_text().splitlines()[0])["model"] == "local-fast"
current = {t.name: t.content_hash for t in TASKS}
[row] = summarise(stored, current)
assert (row.successes, row.runs) == (1, 2)
assert not row.stale
table = render([row])
assert "openai/qwen3-8b-awq" in table
assert "1/2" in table
def test_runs_of_an_older_task_version_are_flagged() -> None:
current = {t.name: t.content_hash for t in TASKS}
rows = summarise([record_json(task_hash="0123456789abcdef")], current)
assert rows[0].stale
assert "(old)" in render(rows)
def test_models_served_differently_are_kept_apart() -> None:
current = {t.name: t.content_hash for t in TASKS}
rows = summarise(
[record_json(note="vLLM, AWQ"), record_json(note="llama.cpp, Q4_K_M")], current
)
assert len(rows) == 2
def test_metering_counts_prose_and_cost() -> None:
class Inner:
def __init__(self) -> None:
self.last_usage = {"input": 10, "output": 2}
self.last_cost = 0.5
self.last_model = "local-fast"
def complete(self, events, tools, model, system=None):
return [AssistantText(text="hi", model=model)]
meter = MeteringClient(Inner(), FACTS)
meter.complete([], None, "local-fast")
meter.complete([], None, "local-fast")
assert (meter.calls, meter.prose_turns, meter.cost) == (2, 2, 1.0)
def test_metering_counts_rate_limit_waits_even_on_failure() -> None:
class Inner:
def __init__(self) -> None:
self.last_usage = {"input": 10, "output": 2}
self.last_cost = 0.0
self.last_model = "local-fast"
self.last_rate_limit_wait_s = 0.0
self.calls = 0
def complete(self, events, tools, model, system=None):
self.calls += 1
self.last_rate_limit_wait_s = 20.0
if self.calls == 2:
raise RuntimeError("upstream returned 429")
return [AssistantText(text="hi", model=model)]
meter = MeteringClient(Inner(), FACTS)
meter.complete([], None, "local-fast")
with pytest.raises(RuntimeError):
meter.complete([], None, "local-fast")
assert meter.rate_limit_wait_s == 40.0
def test_a_run_records_its_rate_limit_waits(tmp_path: Path) -> None:
[task] = load_tasks(["write-function"])
client = Scripted(list(SOLVE_SLUGIFY))
client.last_rate_limit_wait_s = 20.0 # every call waited once
record = run_task(task, FACTS, client, Tuning())
assert record.rate_limit_wait_s == 60.0
def test_the_report_leaves_rate_limit_waits_out_of_time() -> None:
current = {t.name: t.content_hash for t in TASKS}
rows = summarise(
[
record_json(wall_s=50.0, rate_limit_wait_s=40.0),
record_json(wall_s=10.0, rate_limit_wait_s=0.0),
],
current,
)
[row] = rows
assert row.mean_wall_s == 10.0
assert row.rate_limited == 1
assert "429 = runs that waited" in render(rows)
def test_results_stored_before_waits_were_recorded_still_load() -> None:
current = {t.name: t.content_hash for t in TASKS}
old = record_json(wall_s=12.0)
del old["rate_limit_wait_s"]
[row] = summarise([old], current)
assert (row.mean_wall_s, row.rate_limited) == (12.0, 0)
def test_runs_with_different_tuning_are_kept_apart() -> None:
current = {t.name: t.content_hash for t in TASKS}
capped = record_json()
capped["tuning"]["executor"]["max_turns"] = 7
rows = summarise([record_json(), capped], current)
assert sorted(row.tuning for row in rows) == ["", "executor.max_turns=7"]
table = render(rows)
assert "executor.max_turns=7" in table
assert "tuning = settings that differ" in table
def test_a_report_at_default_tuning_has_no_tuning_column() -> None:
current = {t.name: t.content_hash for t in TASKS}
table = render(summarise([record_json()], current))
assert "tuning" not in table
nisse-dev/tests/test_canonical.py
+5 -2
View File
@@ -16,7 +16,7 @@ from nisse_dev.canonical import (
from_payload,
mint_call_id,
)
from nisse_dev.state import StateStore
from nisse_dev.state import CURRENT_SCHEMA_VERSION, StateStore
ALL_EVENTS = [
UserMessage(text="use demo", source="matrix:$abc"),
@@ -116,9 +116,12 @@ def test_default_migration_upgrades_a_real_v1_store(tmp_path: Path) -> None:
conn.close()
with StateStore(db) as store:
assert store.schema_version == 2
assert store.schema_version == CURRENT_SCHEMA_VERSION
state = store.get_state()
assert state is not None
assert state.state == "IDLE" # v1 data intact
store.append_event(UserMessage(text="hello", source="m:1"))
assert len(store.read_events()) == 1
# Two migrations were traversed, so both later tables must exist.
store.mark_outcome(1, "good", "arrived via v1 -> v3")
assert store.outcomes()[1] == ("good", "arrived via v1 -> v3")
nisse-dev/tests/test_catalog.py
+202
View File
@@ -0,0 +1,202 @@
"""Runtime model discovery (spec A5/A11/A13).
The point of these tests is that nothing here reads the model's *name*. A
semantic alias like "remote-architecture" is free to point anywhere, so the
route has to come from what LiteLLM says the model is.
"""
import httpx
import pytest
from nisse_dev.catalog import (
CatalogUnavailable,
ModelCatalog,
UnknownModel,
select_client,
)
def entry(alias: str, backing: str, provider: str | None, tools: bool | None = None):
"""One /model/info row, shaped as the live proxy returns it."""
info: dict = {"supports_function_calling": tools}
if provider is not None:
info["litellm_provider"] = provider
return {
"model_name": alias,
"litellm_params": {"model": backing, "api_base": "http://proxy.test/v1"},
"model_info": info,
}
def catalog(*entries) -> ModelCatalog:
return ModelCatalog.from_payload({"data": list(entries)})
# --- parsing -------------------------------------------------------------
def test_facts_are_read_from_the_payload() -> None:
cat = catalog(
entry("claude-haiku-4-5", "anthropic/claude-haiku-4-5", "anthropic", True)
)
facts = cat.facts("claude-haiku-4-5")
assert facts.backing == "anthropic/claude-haiku-4-5"
assert facts.provider == "anthropic"
assert facts.supports_tools is True
assert facts.native_anthropic
def test_provider_falls_back_to_the_backing_prefix() -> None:
# litellm_provider is absent on some deployments; the backing model's
# own "vendor/model" prefix says the same thing.
cat = catalog(entry("mystery", "anthropic/claude-opus-5", None))
assert cat.facts("mystery").provider == "anthropic"
def test_unnamed_entries_are_skipped() -> None:
cat = ModelCatalog.from_payload({"data": [{"litellm_params": {"model": "x"}}]})
assert cat.aliases() == []
def test_missing_tool_support_is_unknown_not_false() -> None:
# LiteLLM reports null for models it has no capability data on. That is
# not the same as declaring the model tool-less, and the startup warning
# depends on the difference.
cat = catalog(entry("local-fast", "openai/qwen3-8b-awq", "openai", None))
assert cat.facts("local-fast").supports_tools is None
# --- route selection by provider, never by name --------------------------
def test_a_semantic_alias_routes_by_its_provider() -> None:
# The whole feature: nothing in "remote-architecture" says Anthropic.
cat = catalog(entry("remote-architecture", "anthropic/claude-opus-5", "anthropic"))
normalized, passthrough = object(), object()
chosen = select_client("remote-architecture", normalized, passthrough, cat)
assert chosen is passthrough
def test_the_same_alias_reroutes_when_it_is_repointed() -> None:
# A13: provisionally accept LiteLLM's normalization for a vendor with no
# native adapter, rather than sending it Anthropic-shaped bodies.
cat = catalog(entry("remote-architecture", "openai/gpt-6-astra", "openai"))
normalized, passthrough = object(), object()
assert (
select_client("remote-architecture", normalized, passthrough, cat) is normalized
)
def test_a_name_that_merely_mentions_a_vendor_is_not_that_vendor() -> None:
cat = catalog(entry("claude-clone-9", "openai/some-clone", "openai"))
normalized, passthrough = object(), object()
assert select_client("claude-clone-9", normalized, passthrough, cat) is normalized
def test_local_models_take_the_normalized_route() -> None:
cat = catalog(
entry("local-coder", "openai/qwen3-14b", "openai"),
entry("local-fast", "openai/qwen3-8b-awq", "openai"),
)
normalized, passthrough = object(), object()
for alias in ("local-coder", "local-fast"):
assert select_client(alias, normalized, passthrough, cat) is normalized
# --- misconfiguration is loud --------------------------------------------
def test_an_unknown_model_raises_rather_than_guessing_a_route() -> None:
cat = catalog(entry("local-fast", "openai/qwen3-8b-awq", "openai"))
with pytest.raises(UnknownModel, match="local-fsat"):
select_client("local-fsat", object(), object(), cat)
def test_the_error_lists_what_is_available() -> None:
cat = catalog(
entry("local-fast", "openai/qwen3-8b-awq", "openai"),
entry("claude-opus-5", "anthropic/claude-opus-5", "anthropic"),
)
with pytest.raises(UnknownModel, match="claude-opus-5, local-fast"):
cat.facts("typo")
def test_validate_names_every_bad_role_at_once() -> None:
# One boot should surface the whole list, not make the operator
# rediscover the next typo on the next start.
cat = catalog(entry("local-fast", "openai/qwen3-8b-awq", "openai"))
with pytest.raises(UnknownModel) as caught:
cat.validate(
{
"executor": "local-fast",
"planner": "clause-opus-5",
"delegate": "remote-architecture",
"consultant": "local-fast",
}
)
message = str(caught.value)
assert "planner='clause-opus-5'" in message
assert "delegate='remote-architecture'" in message
assert "executor" not in message # the good ones are not noise
def test_validate_passes_a_whole_good_mapping() -> None:
cat = catalog(
entry("local-fast", "openai/qwen3-8b-awq", "openai"),
entry("claude-opus-5", "anthropic/claude-opus-5", "anthropic"),
)
cat.validate({"executor": "local-fast", "planner": "claude-opus-5"})
# --- an outage is not a misconfiguration ---------------------------------
def test_an_unreachable_proxy_is_reported_as_an_outage() -> None:
def handler(request: httpx.Request) -> httpx.Response:
raise httpx.ConnectError("no route to host")
with pytest.raises(CatalogUnavailable, match="could not reach"):
ModelCatalog.discover(
"http://litellm.test", "sk-test", transport=httpx.MockTransport(handler)
)
def test_a_refused_listing_is_reported_as_an_outage() -> None:
cat = httpx.MockTransport(lambda request: httpx.Response(401, text="bad key"))
with pytest.raises(CatalogUnavailable, match="401"):
ModelCatalog.discover("http://litellm.test", "sk-test", transport=cat)
def test_discovery_asks_the_proxy_with_the_token() -> None:
seen: dict = {}
def handler(request: httpx.Request) -> httpx.Response:
seen["url"] = str(request.url)
seen["auth"] = request.headers.get("Authorization")
return httpx.Response(
200, json={"data": [entry("local-fast", "openai/qwen3-8b-awq", "openai")]}
)
cat = ModelCatalog.discover(
"http://litellm.test/", "sk-test", transport=httpx.MockTransport(handler)
)
assert seen["url"] == "http://litellm.test/model/info"
assert seen["auth"] == "Bearer sk-test"
assert cat.aliases() == ["local-fast"]
# --- what gets pinned ----------------------------------------------------
def test_pinning_records_what_the_alias_resolved_to() -> None:
# An alias may point somewhere else tomorrow; eval history (A16) keyed on
# the alias alone would average across whatever it pointed at.
cat = catalog(entry("remote-architecture", "anthropic/claude-opus-5", "anthropic"))
pinned = cat.pinned({"planner": "remote-architecture"})
assert pinned == {
"planner": {
"alias": "remote-architecture",
"backing": "anthropic/claude-opus-5",
"provider": "anthropic",
}
}
nisse-dev/tests/test_config.py
+308
View File
@@ -0,0 +1,308 @@
import copy
from pathlib import Path
from typing import Any
import pytest
from nisse_dev.config import AgentConfig, ConfigError
# A complete config as the YAML file would give it, with secrets by reference.
COMPLETE: dict[str, Any] = {
"state_db": "~/demo/state.db",
"workspace": "~/demo/work",
"project": "greet",
"llm": {"base_url": "https://litellm.test", "token": "${TEST_LLM_TOKEN}"},
"roles": {
"executor": "claude-haiku-4-5",
"planner": "claude-opus-5",
"delegate": "claude-opus-5",
"consultant": "claude-haiku-4-5",
},
"matrix": {
"homeserver": "https://matrix.test/",
"user_id": "@nisse-dev-testing:test",
"token": "${TEST_MATRIX_TOKEN}",
"room_id": "!room:test",
"operators": ["@don:test"],
},
"gitea": {
"url": "https://gitea.test",
"token": "${TEST_GITEA_TOKEN}",
"repo": "don/demo",
"operators": ["don"],
},
}
SECRETS = {
"TEST_LLM_TOKEN": "sk-x",
"TEST_MATRIX_TOKEN": "syt_x",
"TEST_GITEA_TOKEN": "gt",
}
def tree(**changes: Any) -> dict[str, Any]:
"""COMPLETE with dotted keys changed; a value of None removes the key."""
result = copy.deepcopy(COMPLETE)
for dotted, value in changes.items():
*parents, leaf = dotted.split("__")
node = result
for part in parents:
node = node.setdefault(part, {})
if value is None:
node.pop(leaf, None)
else:
node[leaf] = value
return result
def build(**changes: Any) -> AgentConfig:
return AgentConfig.from_tree(tree(**changes), SECRETS)
def test_complete_config_builds_with_secrets_by_reference() -> None:
config = build()
assert config.project == "greet"
assert config.llm_token == "sk-x"
assert config.gitea_token == "gt"
assert config.roles.planner == "claude-opus-5"
assert config.matrix.room_id == "!room:test"
assert config.matrix.homeserver == "https://matrix.test"
assert config.gitea_repo == "don/demo"
assert config.base_branch == "main"
assert config.tuning.executor.max_turns == 40 # absent section: defaults
def test_paths_expand_the_home_directory() -> None:
config = build()
assert config.workspace.is_absolute()
assert "~" not in str(config.state_db)
def test_every_missing_setting_is_reported_at_once() -> None:
# Failing at first use would surface a missing token twenty minutes into
# a milestone, on the call that mattered.
with pytest.raises(ConfigError) as excinfo:
build(matrix__token=None, gitea__repo=None)
message = str(excinfo.value)
assert "matrix.token" in message
assert "gitea.repo" in message
def test_empty_is_treated_as_missing() -> None:
with pytest.raises(ConfigError, match="gitea.repo"):
build(gitea__repo=" ")
def test_a_reference_to_an_unset_variable_is_an_error() -> None:
# Never an empty string: an empty token would pass for a value and fail
# at the first request.
env = {k: v for k, v in SECRETS.items() if k != "TEST_GITEA_TOKEN"}
with pytest.raises(ConfigError, match=r"gitea.token: references \$\{TEST_GITEA"):
AgentConfig.from_tree(tree(), env)
def test_operator_lists_drop_blanks() -> None:
config = build(matrix__operators=[" @don:test ", "@other:test", " "])
assert config.operators == {"@don:test", "@other:test"}
# A blank entry must not admit an empty identity into an allowlist.
assert "" not in config.operators
def test_an_operator_list_of_only_blanks_is_missing() -> None:
with pytest.raises(ConfigError, match="gitea.operators"):
build(gitea__operators=[" "])
def test_base_branch_is_configurable() -> None:
assert build(gitea__base_branch="trunk").base_branch == "trunk"
def test_max_output_tokens_is_configurable_and_checked() -> None:
assert build(llm__max_output_tokens=8192).max_output_tokens == 8192
with pytest.raises(ConfigError, match="llm.max_output_tokens"):
build(llm__max_output_tokens=0)
def test_redacted_summary_carries_no_secrets() -> None:
summary = build().redacted()
flat = " ".join(summary.values())
for secret in SECRETS.values():
assert secret not in flat
assert summary["project"] == "greet"
def test_tuning_is_read() -> None:
config = build(
tuning={
"executor": {"max_turns": 40, "failure_limit": 3},
"expert": {"consult_budget": 5},
"delegation": {"max_turns": 20, "max_tokens": 100000, "wall_clock_s": 600},
"tools": {"command_timeout_s": 300, "max_output_chars": 2000},
}
)
tuning = config.tuning
assert (tuning.executor.max_turns, tuning.executor.failure_limit) == (40, 3)
assert tuning.expert.consult_budget == 5
assert tuning.delegation.max_tokens == 100_000
assert tuning.delegation.wall_clock_s == 600.0
assert tuning.tools.max_output_chars == 2000
def test_a_partly_given_tuning_section_keeps_the_other_defaults() -> None:
tuning = build(tuning={"executor": {"max_turns": 40}}).tuning
assert tuning.executor.failure_limit == 2
def test_every_problem_is_reported_at_once() -> None:
# A misspelt knob that was silently ignored would look exactly like a knob
# with no effect, so unknown keys are errors.
bad = tree(project=None, llm_extra=1)
bad["tuning"] = {
"executor": {"max_turn": 40, "failure_limit": 0},
"expert": {"consult_budget": True},
"bogus": {"x": 1},
}
with pytest.raises(ConfigError) as excinfo:
AgentConfig.from_tree(bad, SECRETS)
message = str(excinfo.value)
for expected in (
"project",
"llm_extra: unknown setting",
"tuning.executor.max_turn: unknown setting",
"tuning.executor.failure_limit: must be a positive number",
"tuning.expert.consult_budget: must be a positive number",
"tuning.bogus: unknown section",
):
assert expected in message
def test_a_whole_number_is_required_where_one_is_meant() -> None:
with pytest.raises(ConfigError, match="tuning.executor.max_turns"):
build(tuning={"executor": {"max_turns": 2.5}})
# --- the file --------------------------------------------------------------
FILE = """
state_db: ~/demo/state.db
workspace: ~/demo/work
project: greet
llm: {base_url: "https://litellm.test", token: "${TEST_LLM_TOKEN}"}
roles:
executor: local-fast
planner: claude-opus-5
delegate: claude-opus-5
consultant: claude-haiku-4-5
matrix:
homeserver: https://matrix.test
user_id: "@nisse-dev-testing:test"
token: ${TEST_MATRIX_TOKEN}
room_id: "!room:test"
operators: ["@don:test"]
gitea:
url: https://gitea.test
token: ${TEST_GITEA_TOKEN}
repo: don/demo
operators: [don]
"""
def write(directory: Path, text: str = FILE) -> Path:
path = directory / "nisse-dev.yaml"
path.write_text(text)
return path
def test_a_file_is_read(tmp_path: Path) -> None:
config = AgentConfig.from_file(write(tmp_path), SECRETS)
assert config.roles.executor == "local-fast"
assert config.config_file == tmp_path / "nisse-dev.yaml"
def test_invalid_yaml_is_reported(tmp_path: Path) -> None:
with pytest.raises(ConfigError, match="not valid YAML"):
AgentConfig.from_file(write(tmp_path, "roles: [unclosed"), SECRETS)
def test_load_uses_nisse_config(tmp_path: Path) -> None:
path = write(tmp_path)
assert AgentConfig.load({**SECRETS, "NISSE_CONFIG": str(path)}).project == "greet"
def test_load_defaults_to_the_working_directory(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
write(tmp_path)
monkeypatch.chdir(tmp_path)
assert AgentConfig.load(SECRETS).project == "greet"
def test_load_without_a_file_says_how_to_make_one(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.chdir(tmp_path)
with pytest.raises(ConfigError, match="nisse-dev.example.yaml"):
AgentConfig.load(SECRETS)
def test_the_example_file_is_valid() -> None:
# The example documents every knob; it must not drift out of loading.
example = Path(__file__).resolve().parents[2] / "nisse-dev.example.yaml"
env = {
"NISSE_LLM_TOKEN": "sk-x",
"NISSE_MATRIX_TOKEN": "syt_x",
"NISSE_GITEA_TOKEN": "gt",
}
assert AgentConfig.from_file(example, env).tuning.executor.max_turns == 40
# --- nothing that carries secrets may sit inside the clone -----------------
def test_state_db_and_workspace_are_separate() -> None:
# A2: build artefacts must not share a volume with agent state.
config = build()
assert config.state_db.parent != config.workspace
def test_state_db_inside_the_clone_is_refused(tmp_path: Path) -> None:
# git add -A would commit it, and the push would publish the agent's
# own state alongside the project.
work = tmp_path / "project"
with pytest.raises(ConfigError, match="inside the workspace"):
build(workspace=str(work), state_db=str(work / "state.db"))
def test_env_file_inside_the_clone_is_refused(tmp_path: Path) -> None:
# The severe case: the agent would commit and push its own credentials
# as a side effect of working correctly.
work = tmp_path / "project"
with pytest.raises(ConfigError, match="NISSE_ENV"):
AgentConfig.from_tree(
tree(workspace=str(work), state_db=str(tmp_path / "state.db")),
{**SECRETS, "NISSE_ENV": str(work / ".env")},
)
def test_a_config_file_inside_the_clone_is_refused(tmp_path: Path) -> None:
work = tmp_path / "project"
work.mkdir()
text = FILE.replace("workspace: ~/demo/work", f"workspace: {work}")
with pytest.raises(ConfigError, match="NISSE_CONFIG"):
AgentConfig.from_file(write(work, text), SECRETS)
def test_sibling_paths_are_fine(tmp_path: Path) -> None:
config = AgentConfig.from_tree(
tree(workspace=str(tmp_path / "project"), state_db=str(tmp_path / "state.db")),
{**SECRETS, "NISSE_ENV": str(tmp_path / ".env")},
)
assert config.workspace.name == "project"
def test_context_tuning_is_read_and_checked() -> None:
config = build(tuning={"context": {"keep_recent_turns": 4, "safety_margin": 0.2}})
assert config.tuning.context.keep_recent_turns == 4
assert config.tuning.context.safety_margin == 0.2
assert build().tuning.context.keep_recent_turns == 6
with pytest.raises(ConfigError, match="safety_margin: must be less than 1"):
build(tuning={"context": {"safety_margin": 1}})
nisse-dev/tests/test_context.py
+342
View File
@@ -0,0 +1,342 @@
"""The context budget (spec A10)."""
import io
from pathlib import Path
from typing import Any
import pytest
from nisse_dev.canonical import (
AssistantText,
CanonicalEvent,
GateOpened,
PlanSubmitted,
StoredEvent,
ToolCall,
ToolResult,
UserMessage,
)
from nisse_dev.catalog import ModelCatalog, UnknownLimits
from nisse_dev.context import (
Budget,
BudgetedClient,
ContextOverflow,
estimate_tokens,
fit,
head_and_tail,
)
from nisse_dev.executor import LLMExecutor
from nisse_dev.llm import RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
def stored(events: list[CanonicalEvent]) -> list[StoredEvent]:
return [StoredEvent(id=i + 1, ts="t", event=e) for i, e in enumerate(events)]
def turn(n: int, output: str) -> list[CanonicalEvent]:
"""One executor turn: a command and its (possibly long) output."""
return [
ToolCall(call_id=f"c{n}", tool="run_command", arguments={"command": "make"}),
ToolResult(call_id=f"c{n}", result=output, success=True),
]
def transcript(turns: int, output_chars: int) -> list[StoredEvent]:
events: list[CanonicalEvent] = [UserMessage(text="start milestone 1", source="t")]
for n in range(turns):
events += turn(n, f"output {n} " + "x" * output_chars)
return stored(events)
def budget(window: int, keep: int = 2) -> Budget:
return Budget(
window=window, max_output=500, keep_recent_turns=keep, safety_margin=0.1
)
def results(events: list[StoredEvent]) -> dict[str, str]:
return {
e.event.call_id: e.event.result
for e in events
if isinstance(e.event, ToolResult)
}
# --- keeping the end of long output ---------------------------------------
def test_head_and_tail_keeps_the_failure_summary_at_the_end() -> None:
output = (
"exit code: 1\nstdout:\n"
+ "collected 400 items\n" * 500
+ "FAILED tests/test_app.py::test_greet - AssertionError\n"
+ "==== 1 failed, 399 passed ====\n"
)
kept = head_and_tail(output, 1000)
assert len(kept) < 1100
assert kept.startswith("exit code: 1") # the head keeps the exit code
assert "FAILED tests/test_app.py::test_greet" in kept
assert "1 failed, 399 passed" in kept
assert "characters omitted" in kept
def test_short_text_is_untouched() -> None:
assert head_and_tail("fine", 1000) == "fine"
# --- fitting, step by step -------------------------------------------------
def test_a_transcript_that_fits_is_sent_unchanged() -> None:
events = transcript(turns=3, output_chars=100)
assert fit(events, "system", None, budget(100_000)) == events
def test_step_one_cuts_older_tool_results_and_keeps_recent_ones_whole() -> None:
events = transcript(turns=6, output_chars=3000)
fitted = fit(events, "system", None, budget(4000, keep=2))
kept = results(fitted)
assert len(kept) == 6 # nothing dropped yet
for older in ("c0", "c1", "c2", "c3"):
assert "characters omitted" in kept[older]
for recent in ("c4", "c5"):
assert "characters omitted" not in kept[recent]
def test_step_two_drops_the_oldest_turns_and_says_so() -> None:
events = transcript(turns=20, output_chars=3000)
fitted = fit(events, "system", None, budget(4000, keep=2))
first = fitted[0].event
assert isinstance(first, UserMessage)
assert "omitted to fit the context window" in first.text
kept = results(fitted)
assert "c0" not in kept # the oldest went
assert {"c18", "c19"} <= kept.keys() # the recent turns stayed
assert "characters omitted" not in kept["c19"]
# Every surviving result still has its call: a provider rejects an
# answer to a call it never saw.
calls = {e.event.call_id for e in fitted if isinstance(e.event, ToolCall)}
assert kept.keys() <= calls
assert estimate_tokens(fitted, "system", None) <= budget(4000).for_prompt
def test_step_three_refuses_what_cannot_fit() -> None:
# Even the two most recent turns are bigger than the whole window.
events = transcript(turns=3, output_chars=20_000)
with pytest.raises(ContextOverflow, match="last 2 turns"):
fit(events, "system", None, budget(2000, keep=2))
def test_a_huge_system_prompt_alone_overflows() -> None:
events = transcript(turns=1, output_chars=10)
with pytest.raises(ContextOverflow):
fit(events, "s" * 50_000, None, budget(4096))
def test_the_output_ceiling_and_margin_come_out_of_the_window() -> None:
b = Budget(window=4096, max_output=1024, keep_recent_turns=6, safety_margin=0.1)
assert b.for_prompt == int(4096 * 0.9) - 1024
# --- the client wrapper ----------------------------------------------------
class RecordingClient:
def __init__(self, reported_input: int = 0) -> None:
self.calls: list[dict[str, Any]] = []
self.last_usage = {"input": reported_input, "output": 5}
def complete(self, events, tools, model, system=None, *, max_tokens=None, **kwargs):
self.calls.append({"events": events, "max_tokens": max_tokens, **kwargs})
return [AssistantText(text="ok", model=model)]
def test_the_ceiling_is_the_smaller_of_config_and_model() -> None:
inner = RecordingClient()
client = BudgetedClient(inner, lambda m: (4096, 1024), max_output_tokens=16384)
client.complete(transcript(1, 10), None, "llama3.2")
assert inner.calls[0]["max_tokens"] == 1024
client = BudgetedClient(inner, lambda m: (200_000, 64_000), max_output_tokens=16384)
client.complete(transcript(1, 10), None, "claude-haiku-4-5")
assert inner.calls[1]["max_tokens"] == 16384
def test_the_wrapper_sends_the_fitted_events() -> None:
inner = RecordingClient()
client = BudgetedClient(
inner, lambda m: (4000, 500), keep_recent_turns=2, safety_margin=0.1
)
client.complete(transcript(20, 3000), None, "m", system="system")
sent = inner.calls[0]["events"]
assert "omitted to fit" in sent[0].event.text
def test_reported_usage_only_ever_raises_the_estimate() -> None:
events = transcript(1, 10)
estimate = estimate_tokens(events, None, None)
under = RecordingClient(reported_input=estimate * 2) # we undercounted
client = BudgetedClient(under, lambda m: (100_000, None))
client.complete(events, None, "m")
assert client._factors["m"] == pytest.approx(2.0, rel=0.05)
over = RecordingClient(reported_input=estimate // 2) # we overcounted
under_then_over = BudgetedClient(over, lambda m: (100_000, None))
under_then_over._factors["m"] = 2.0
under_then_over.complete(events, None, "m")
assert under_then_over._factors["m"] == 2.0 # never loosened
def test_last_usage_passes_through() -> None:
client = BudgetedClient(RecordingClient(reported_input=7), lambda m: (100, None))
assert client.last_usage["input"] == 7
# --- the limits come from LiteLLM's catalog --------------------------------
CATALOG = ModelCatalog.from_payload(
{
"data": [
{
"model_name": "claude-haiku-4-5",
"litellm_params": {"model": "anthropic/claude-haiku-4-5"},
"model_info": {"max_input_tokens": 200000, "max_output_tokens": 64000},
},
{
"model_name": "llama3.2",
"litellm_params": {"model": "ollama/llama3.2"},
"model_info": {"max_input_tokens": 4096, "max_output_tokens": 1024},
},
{
"model_name": "undeclared",
"litellm_params": {"model": "ollama/qwen3"},
"model_info": {"max_input_tokens": None},
},
]
}
)
def test_the_catalog_reads_each_models_limits() -> None:
facts = CATALOG.facts("llama3.2")
assert (facts.max_input_tokens, facts.max_output_tokens) == (4096, 1024)
assert CATALOG.facts("undeclared").max_input_tokens is None
def test_a_role_on_a_model_without_a_window_is_refused() -> None:
CATALOG.require_limits({"executor": "llama3.2", "planner": "claude-haiku-4-5"})
with pytest.raises(UnknownLimits, match="executor='undeclared'"):
CATALOG.require_limits({"executor": "undeclared", "planner": "llama3.2"})
# --- the executor sees the approved plan and what came after ---------------
def test_the_executor_sees_only_its_milestone(tmp_path: Path) -> None:
store = StateStore(tmp_path / "state.db", oplog=OpLog("t", io.StringIO()))
for event in (
UserMessage(text="use greet: a long planning conversation", source="t"),
PlanSubmitted(plan="arch plan", model="m"),
UserMessage(text="approve", source="t"),
PlanSubmitted(plan="milestone 1: write greet.py", model="m"),
GateOpened(gate="milestone-1", reason="plan submitted"),
UserMessage(text="start milestone 1", source="t"),
):
store.append_event(event)
class Workspace:
def describe(self) -> str:
return "workspace"
seen: dict[str, Any] = {}
class Client:
def complete(self, events, tools, model, system=None):
seen["events"] = [e.event for e in events]
seen["system"] = system
return [AssistantText(text="done", model=model)]
roles = RoleMap(executor="m", planner="m", delegate="m", consultant="m")
LLMExecutor(Client(), store, roles, Workspace()).run(1)
texts = [getattr(e, "text", None) for e in seen["events"]]
assert "start milestone 1" in texts
assert "use greet: a long planning conversation" not in texts
assert not any(isinstance(e, PlanSubmitted) for e in seen["events"])
# The plan rides in the system prompt, which the budget never drops.
assert "The approved plan:\nmilestone 1: write greet.py" in seen["system"]
assert "arch plan" not in seen["system"]
def test_a_request_that_cannot_fit_stops_the_milestone(tmp_path: Path) -> None:
# Never sent: the executor stops and asks the operator, like any failed
# call, rather than handing a server a prompt it would silently cut.
store = StateStore(tmp_path / "state.db", oplog=OpLog("t", io.StringIO()))
store.append_event(UserMessage(text="start milestone 1", source="t"))
class Workspace:
def describe(self) -> str:
return "workspace " + "w" * 20_000 # a system prompt too big to fit
inner = RecordingClient()
client = BudgetedClient(inner, lambda m: (4096, 1024))
roles = RoleMap(executor="m", planner="m", delegate="m", consultant="m")
outcome = LLMExecutor(client, store, roles, Workspace()).run(1)
assert outcome.status == "escalate"
assert "fit" in outcome.note
assert inner.calls == []
# --- tool_choice is only asked of servers that enforce it -------------------
FINISH = [{"type": "function", "function": {"name": "finish", "parameters": {}}}]
def test_required_is_asked_only_of_models_declared_to_enforce_it() -> None:
# Measured: qwen3-14b under "required" cannot stop and rambles to the
# output ceiling, so a model that is not declared is never asked.
inner = RecordingClient()
client = BudgetedClient(
inner, lambda m: (32768, 8192), enforces_tool_choice=lambda m: m == "local-fast"
)
client.complete(transcript(1, 10), FINISH, "local-fast")
client.complete(transcript(1, 10), FINISH, "local-coder")
client.complete(transcript(1, 10), None, "local-fast")
assert [c["require_tool_call"] for c in inner.calls] == [True, False, False]
def test_the_catalog_reads_supports_tool_choice() -> None:
catalog = ModelCatalog.from_payload(
{
"data": [
{"model_name": "a", "model_info": {"supports_tool_choice": True}},
{"model_name": "b", "model_info": {"supports_tool_choice": False}},
{"model_name": "c", "model_info": {}},
]
}
)
assert [catalog.facts(m).supports_tool_choice for m in "abc"] == [True, False, None]
def test_a_fallback_answer_is_recognised() -> None:
facts = ModelCatalog.from_payload(
{
"data": [
{
"model_name": "claude-haiku-4-5",
"litellm_params": {"model": "anthropic/claude-haiku-4-5"},
}
]
}
).facts("claude-haiku-4-5")
assert facts.answered("claude-haiku-4-5") # LiteLLM's alias
assert facts.answered("claude-haiku-4-5-20251001") # the provider's own name
assert facts.answered(None) # nothing reported
assert not facts.answered("claude-sonnet-5") # somebody else answered
nisse-dev/tests/test_delegation.py
+387 -1
View File
@@ -4,6 +4,7 @@ from typing import Any
import pytest
from nisse_dev.cancel import STOPPED_RESULT, Cancel
from nisse_dev.canonical import (
AssistantText,
CanonicalEvent,
@@ -11,14 +12,17 @@ from nisse_dev.canonical import (
DelegationStarted,
StoredEvent,
ToolCall,
ToolResult,
)
from nisse_dev.delegation import (
DELEGATE_TOOLS,
DELEGATION_INTENT,
Caps,
Delegation,
delegation_reconciler,
)
from nisse_dev.git import GitRepo
from nisse_dev.llm import LLMError
from nisse_dev.llm import BudgetExhausted, LLMError, UpstreamUnavailable
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
from nisse_dev.tools import Workspace
@@ -131,6 +135,46 @@ def test_abort_discards_the_branch_and_its_work(
assert not (repo.root / "half-done.py").exists()
def test_stop_discards_the_branch_and_settles_the_intent(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# A4 stop is treated like a failure, not like an outage: the operator
# stopped it for a reason, so nothing is kept for resuming.
cancel = Cancel()
client = ScriptedClient(
[
[call("write_file", path="half-done.py", content="oops")],
[call("list_dir", path=".")],
]
)
original = client.complete
def complete_then_stop(*args: Any, **kwargs: Any) -> list[CanonicalEvent]:
produced = original(*args, **kwargs)
if len(client.calls) == 2:
cancel.set() # the operator says stop during turn 2's model call
return produced
client.complete = complete_then_stop # type: ignore[method-assign]
result = build(client, store, ws, repo, cancel=cancel).run(
brief="b", branch="deleg-stop"
)
assert result.status == "stopped"
assert not result.usable
assert repo.current_branch() == "main"
assert not repo.branch_exists("deleg-stop")
assert not (repo.root / "half-done.py").exists()
assert store.pending_intents() == []
# The unrun call still has a result, so the transcript renders.
skipped = [
e.event
for e in store.read_events()
if isinstance(e.event, ToolResult) and e.event.result == STOPPED_RESULT
]
assert len(skipped) == 1
def test_turn_cap_is_hard(store: StateStore, ws: Workspace, repo: GitRepo) -> None:
forever = [[call("list_dir", path=".")] for _ in range(20)]
client = ScriptedClient(forever)
@@ -302,3 +346,345 @@ def test_failure_may_be_reported_without_doing_work(
assert result.status == "failed"
assert not result.usable
assert len(client.calls) == 1 # believed immediately, not challenged
# --- A3 step 5: the resume rule -----------------------------------------
def pending_delegation_intent(store: StateStore, branch: str = "d") -> Any:
store.begin_intent(
DELEGATION_INTENT,
{"branch": branch, "base": "main", "brief": "finish it"},
cause="test",
)
return store.pending_intents()[-1]
def test_intent_is_recorded_and_settled_by_a_completed_delegation(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = ScriptedClient(
[[call("write_file", path="a.py", content="x")], [hand_back()]]
)
build(client, store, ws, repo).run(brief="b", branch="d")
# Nothing left in flight: an aborted delegation is a finished one too.
assert store.pending_intents() == []
def test_aborted_delegation_also_settles_its_intent(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = ScriptedClient([[hand_back(succeeded=False, changed="cannot")]])
build(client, store, ws, repo).run(brief="b", branch="d")
assert store.pending_intents() == []
def test_clean_cut_after_a_tool_result_is_resumable(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
repo.checkout_new_branch("d")
(repo.root / "partial.py").write_text("x")
repo.commit_all("partial work")
store.append_event(
DelegationStarted(brief="b", role="delegate", model="m", caps={})
)
store.append_event(ToolCall(call_id="c", tool="write_file", arguments={}))
store.append_event(ToolResult(call_id="c", result="wrote", success=True))
intent = pending_delegation_intent(store)
assert delegation_reconciler(store, repo)(intent) is True
def test_dangling_tool_call_gates(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# The effect of the call is unknown and not verifiable by inspection.
repo.checkout_new_branch("d")
store.append_event(
DelegationStarted(brief="b", role="delegate", model="m", caps={})
)
store.append_event(ToolCall(call_id="c", tool="run_command", arguments={}))
intent = pending_delegation_intent(store)
assert delegation_reconciler(store, repo)(intent) is None
def test_missing_branch_means_nothing_started(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
store.append_event(
DelegationStarted(brief="b", role="delegate", model="m", caps={})
)
store.append_event(ToolResult(call_id="c", result="ok", success=True))
intent = pending_delegation_intent(store, branch="never-created")
assert delegation_reconciler(store, repo)(intent) is False
def test_a_finished_delegation_settles_rather_than_resumes(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
repo.checkout_new_branch("d")
store.append_event(
DelegationStarted(brief="b", role="delegate", model="m", caps={})
)
store.append_event(
DelegationCompleted(
handback="done",
artifacts=[],
tokens=0,
duration_s=1.0,
exit_status="handback",
)
)
intent = pending_delegation_intent(store)
assert delegation_reconciler(store, repo)(intent) is True
def test_resume_continues_the_same_delegation(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# Interrupt after one turn, then resume: no second delegation_started,
# because A10 defines a session as the delegation, not the process.
repo.checkout_new_branch("d")
store.append_event(
DelegationStarted(brief="b", role="delegate", model="m", caps={})
)
store.append_event(ToolResult(call_id="c", result="wrote a.py", success=True))
intent = pending_delegation_intent(store)
client = ScriptedClient(
[[call("write_file", path="b.py", content="x")], [hand_back()]]
)
result = build(client, store, ws, repo).resume(intent)
assert result.usable
starts = [e for e in store.read_events() if isinstance(e.event, DelegationStarted)]
assert len(starts) == 1
assert store.pending_intents() == []
assert (repo.root / "b.py").exists()
def test_resume_onto_a_missing_branch_fails_cleanly(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
intent = pending_delegation_intent(store, branch="gone")
result = build(ScriptedClient([]), store, ws, repo).resume(intent)
assert result.status == "failed"
assert not result.usable
assert store.pending_intents() == [] # not left dangling for the next boot
# --- boot recovery sweep ------------------------------------------------
def test_recover_resumes_a_clean_cut_with_a_notice(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
repo.checkout_new_branch("d")
store.append_event(
DelegationStarted(brief="b", role="delegate", model="m", caps={})
)
store.append_event(ToolResult(call_id="c", result="wrote a.py", success=True))
pending_delegation_intent(store)
notices: list[str] = []
client = ScriptedClient(
[[call("write_file", path="b.py", content="x")], [hand_back()]]
)
result = build(client, store, ws, repo).recover(notices.append)
assert result is not None and result.usable
# A3: a notice, not a question — a clean cut needs no operator decision.
assert notices and "resuming the delegation" in notices[0]
def test_recover_settles_a_finished_delegation_without_resuming(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
repo.checkout_new_branch("d")
store.append_event(
DelegationStarted(brief="b", role="delegate", model="m", caps={})
)
store.append_event(
DelegationCompleted(
handback="done",
artifacts=[],
tokens=0,
duration_s=1.0,
exit_status="handback",
)
)
pending_delegation_intent(store)
client = ScriptedClient([])
assert build(client, store, ws, repo).recover(lambda _m: None) is None
assert store.pending_intents() == []
assert client.calls == [] # the model was never asked
def test_recover_discards_an_intent_whose_branch_never_existed(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
store.append_event(ToolResult(call_id="c", result="ok", success=True))
pending_delegation_intent(store, branch="never-made")
assert build(ScriptedClient([]), store, ws, repo).recover(lambda _m: None) is None
assert store.pending_intents() == []
def test_recover_leaves_an_unclean_cut_for_the_boot_gate(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
repo.checkout_new_branch("d")
store.append_event(
DelegationStarted(brief="b", role="delegate", model="m", caps={})
)
store.append_event(ToolCall(call_id="c", tool="run_command", arguments={}))
pending_delegation_intent(store)
assert build(ScriptedClient([]), store, ws, repo).recover(lambda _m: None) is None
# Still pending, so reconcile_on_boot gates rather than blind-resuming.
assert [i.kind for i in store.pending_intents()] == [DELEGATION_INTENT]
# --- the token cap, which A13 calls hard and was decoration -------------
class MeteredClient(ScriptedClient):
"""A client that reports usage, as the real ones do."""
def __init__(self, turns: list, per_call: int = 1000) -> None:
super().__init__(turns)
self._per_call = per_call
self.last_usage: dict[str, int] = {}
def complete(self, *args: Any, **kwargs: Any):
produced = super().complete(*args, **kwargs)
self.last_usage = {"input": self._per_call, "output": self._per_call}
return produced
def test_token_cap_stops_a_delegation(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# An unbounded delegation can exhaust a daily budget and take external
# models away until it resets, so the cap has to be real.
client = MeteredClient(
[[call("list_dir", path=".")] for _ in range(20)], per_call=600
)
result = build(client, store, ws, repo, caps=Caps(turns=20, tokens=2000)).run(
brief="b", branch="d"
)
assert result.status == "capped"
assert "token cap of 2000" in result.note
# 600 in + 600 out per call: stops on the second turn, not the twentieth.
assert len(client.calls) == 2
def test_tokens_spent_are_reported_not_left_at_zero(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = MeteredClient(
[[call("write_file", path="a.py", content="x")], [hand_back()]], per_call=500
)
result = build(client, store, ws, repo).run(brief="b", branch="d")
assert result.usable
assert result.tokens == 2000 # two calls, 1000 each
completed = next(
e.event for e in store.read_events() if isinstance(e.event, DelegationCompleted)
)
# A16 evaluates cost against outcome, which needs a real number.
assert completed.tokens == 2000
def test_a_client_without_usage_still_works(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# Scripted and third-party clients need not report usage; the cap simply
# does not bind rather than crashing the delegation.
client = ScriptedClient(
[[call("write_file", path="a.py", content="x")], [hand_back()]]
)
assert build(client, store, ws, repo).run(brief="b", branch="d").usable
def test_budget_exhaustion_preserves_the_work_for_later(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
"""An outage is not a bad delegate.
The branch may hold perfectly good work, so it is kept and the intent
left pending — the next boot treats it exactly like a crash mid-delegation
and resumes if the cut was clean. Discarding would throw away work
because the budget ran out.
"""
class BrokeClient(ScriptedClient):
def complete(self, *args: Any, **kwargs: Any):
if self.calls:
raise BudgetExhausted("daily budget exhausted")
return super().complete(*args, **kwargs)
client = BrokeClient([[call("write_file", path="progress.py", content="real")]])
result = build(client, store, ws, repo).run(brief="b", branch="d")
assert result.status == "interrupted"
assert result.interrupted and not result.usable
# The work survives, on its branch.
assert repo.current_branch() == "d"
assert (repo.root / "progress.py").exists()
# And the intent stays pending, which is what A3's resume rule keys on.
assert [i.kind for i in store.pending_intents()] == [DELEGATION_INTENT]
def test_an_ordinary_failure_still_discards(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# Only external interruptions are preserved; a delegate that made a mess
# is still cleaned up (A13).
client = ScriptedClient([[hand_back(succeeded=False, changed="cannot")]])
result = build(client, store, ws, repo).run(brief="b", branch="d")
assert result.status == "failed"
assert not result.interrupted
assert repo.current_branch() == "main"
assert store.pending_intents() == []
def test_a_network_outage_also_preserves_the_work(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# Same reasoning as an exhausted budget: the branch may be fine, and the
# outage says nothing about the quality of the work on it.
class FlakyClient(ScriptedClient):
def complete(self, *args: Any, **kwargs: Any):
if self.calls:
raise UpstreamUnavailable("LLM request failed: connection refused")
return super().complete(*args, **kwargs)
client = FlakyClient([[call("write_file", path="progress.py", content="real")]])
result = build(client, store, ws, repo).run(brief="b", branch="d")
assert result.status == "interrupted"
assert (repo.root / "progress.py").exists()
assert [i.kind for i in store.pending_intents()] == [DELEGATION_INTENT]
def test_a_bad_request_is_not_given_the_benefit_of_the_doubt(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# A 4xx that is not a rate limit means we sent something wrong; treating
# that as an outage would preserve branches and hide a real bug.
class BadRequestClient(ScriptedClient):
def complete(self, *args: Any, **kwargs: Any):
raise LLMError("LLM request returned 400: malformed tool schema")
result = build(BadRequestClient([]), store, ws, repo).run(brief="b", branch="d")
assert result.status == "failed"
assert not result.interrupted
nisse-dev/tests/test_dev_main.py
+19 -5
View File
@@ -1,10 +1,24 @@
from pathlib import Path
import pytest
from nisse_dev.main import main
def test_main_prints_greeting(capsys: pytest.CaptureFixture[str]) -> None:
main()
assert (
capsys.readouterr().out == "Hello from nisse-dev, the house spirit is awake.\n"
)
def test_main_refuses_to_start_without_a_config_file(
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
tmp_path: Path,
) -> None:
# Run from an empty directory, and point NISSE_ENV at a file that does not
# exist: the repository's .env holds live credentials, and a test must
# never be one cwd away from starting a real agent.
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("NISSE_CONFIG", raising=False)
monkeypatch.setenv("NISSE_ENV", str(tmp_path / "absent.env"))
assert main() == 2
err = capsys.readouterr().err
assert "cannot start" in err
assert "nisse-dev.example.yaml" in err
nisse-dev/tests/test_evals.py
+129
View File
@@ -0,0 +1,129 @@
import io
import json
from pathlib import Path
import pytest
from nisse_dev.canonical import (
AssistantText,
DelegationCompleted,
DelegationStarted,
)
from nisse_dev.evals import UNMARKED, collect, summarise, to_jsonl
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
@pytest.fixture
def store(tmp_path: Path) -> StateStore:
return StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
def record_pair(
store: StateStore,
*,
brief: str = "add greet",
model: str = "claude-haiku-4-5",
role: str = "delegate",
status: str = "handback",
) -> int:
store.append_event(
DelegationStarted(brief=brief, role=role, model=model, caps={"turns": 5})
)
store.append_event(
DelegationCompleted(
handback="what changed: greet.py",
artifacts=["greet.py"],
tokens=1200,
duration_s=9.5,
exit_status=status,
)
)
return store.read_events()[-1].id
def test_pairs_a_start_with_its_completion(store: StateStore) -> None:
record_pair(store)
[record] = collect(store)
assert record.brief == "add greet"
assert record.model == "claude-haiku-4-5"
assert record.exit_status == "handback"
assert record.artifacts == ["greet.py"]
assert record.verdict == UNMARKED
def test_unpaired_start_is_skipped(store: StateStore) -> None:
# A crash mid-delegation leaves a start with no completion; a half-formed
# record would pollute the set A16 exists to make trustworthy.
store.append_event(
DelegationStarted(brief="died", role="delegate", model="m", caps={})
)
assert collect(store) == []
def test_unrelated_events_between_the_pair_are_ignored(store: StateStore) -> None:
store.append_event(
DelegationStarted(brief="b", role="delegate", model="m", caps={})
)
store.append_event(AssistantText(text="working", model="m"))
store.append_event(
DelegationCompleted(
handback="done",
artifacts=[],
tokens=1,
duration_s=1.0,
exit_status="handback",
)
)
assert len(collect(store)) == 1
def test_verdicts_attach_to_the_completion(store: StateStore) -> None:
event_id = record_pair(store)
store.mark_outcome(event_id, "good", "clean handback, tests passed")
[record] = collect(store)
assert record.verdict == "good"
assert "tests passed" in record.note
def test_a_verdict_can_be_revised(store: StateStore) -> None:
# Judgements get revised; the log they describe does not.
event_id = record_pair(store)
store.mark_outcome(event_id, "good")
store.mark_outcome(event_id, "bad", "regressed later")
[record] = collect(store)
assert record.verdict == "bad"
assert record.note == "regressed later"
def test_consults_and_delegations_share_the_set(store: StateStore) -> None:
record_pair(store, role="delegate")
record_pair(store, role="consultant", brief="which approach?")
roles = [r.role for r in collect(store)]
assert roles == ["delegate", "consultant"]
def test_collect_can_start_after_a_marker(store: StateStore) -> None:
record_pair(store, brief="old")
marker = store.read_events()[-1].id
record_pair(store, brief="new")
assert [r.brief for r in collect(store, after=marker)] == ["new"]
def test_jsonl_round_trips(store: StateStore) -> None:
record_pair(store)
line = to_jsonl(collect(store))
assert json.loads(line)["model"] == "claude-haiku-4-5"
def test_summary_groups_by_model(store: StateStore) -> None:
good = record_pair(store, model="haiku")
record_pair(store, model="haiku", status="capped")
record_pair(store, model="opus")
store.mark_outcome(good, "good")
summary = summarise(collect(store))
assert summary["haiku"]["total"] == 2
assert summary["haiku"]["good"] == 1
assert summary["haiku"][UNMARKED] == 1
assert summary["opus"]["total"] == 1
nisse-dev/tests/test_executor.py
+177 -24
View File
@@ -4,7 +4,13 @@ from typing import Any
import pytest
from nisse_dev.canonical import AssistantText, CanonicalEvent, StoredEvent, ToolCall
from nisse_dev.canonical import (
AssistantText,
CanonicalEvent,
StoredEvent,
ToolCall,
ToolResult,
)
from nisse_dev.executor import LLMExecutor
from nisse_dev.llm import LLMError, RoleMap
from nisse_dev.oplog import OpLog
@@ -33,7 +39,7 @@ class ScriptedClient:
) -> list[CanonicalEvent]:
self.calls.append({"tools": tools, "model": model, "system": system})
if not self._turns:
return [AssistantText(text="done", model=model)]
return [finish(call_id=f"fin-{len(self.calls)}")]
return self._turns.pop(0)
@@ -46,6 +52,10 @@ def call(tool: str, call_id: str = "c1", **arguments: Any) -> ToolCall:
return ToolCall(call_id=call_id, tool=tool, arguments=arguments)
def finish(summary: str = "done", call_id: str = "fin") -> ToolCall:
return ToolCall(call_id=call_id, tool="finish", arguments={"summary": summary})
@pytest.fixture
def store(tmp_path: Path) -> StateStore:
return StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
@@ -66,15 +76,16 @@ def test_tool_calls_are_executed_and_results_logged(
client = ScriptedClient(
[
[call("write_file", path="app.py", content="print(1)\n")],
[AssistantText(text="wrote the file", model="llama3.2")],
[finish("wrote the file")],
]
)
outcome = build(client, store, ws).run(milestone=1)
assert outcome.status == "completed"
assert outcome.note == "wrote the file"
assert (ws.root / "app.py").read_text() == "print(1)\n"
kinds = [type(e.event).__name__ for e in store.read_events()]
assert kinds == ["ToolCall", "ToolResult", "AssistantText"]
assert kinds == ["ToolCall", "ToolResult", "ToolCall", "ToolResult"]
def test_uses_executor_role_and_reduced_manifest(
@@ -84,50 +95,99 @@ def test_uses_executor_role_and_reduced_manifest(
build(client, store, ws).run(milestone=2)
assert client.calls[0]["model"] == "llama3.2" # never the planner
names = [t["function"]["name"] for t in client.calls[0]["tools"]]
assert names == ["read_file", "write_file", "list_dir", "run_command"]
assert names == ["read_file", "write_file", "list_dir", "run_command", "finish"]
assert "milestone 2" in client.calls[0]["system"]
def test_no_tool_call_after_work_ends_the_milestone(
store: StateStore, ws: Workspace
) -> None:
def test_prose_no_longer_ends_the_milestone(store: StateStore, ws: Workspace) -> None:
# Only finish ends a milestone. Prose breaks the tool contract: the model
# is told once, and here it gets back to work.
client = ScriptedClient(
[
[call("write_file", path="a.py", content="x")],
[AssistantText(text="all steps done", model="m")],
[finish()],
]
)
assert build(client, store, ws).run(milestone=1).status == "completed"
assert "Every reply must be a tool call" in client.calls[2]["system"]
def test_completion_claimed_without_any_work_is_challenged(
def test_persisting_in_prose_escalates(store: StateStore, ws: Workspace) -> None:
prose = [AssistantText(text="let me just chat", model="m")]
client = ScriptedClient([prose, prose])
outcome = build(client, store, ws).run(milestone=1)
assert outcome.status == "escalate"
assert "prose" in outcome.note
def test_finishing_without_any_work_is_refused_once(
store: StateStore, ws: Workspace
) -> None:
# A model that answers with prose on turn one has done nothing; believing
# it would record an empty milestone as a success.
# A model that finishes on turn one has done nothing; believing it would
# record an empty milestone as a success.
client = ScriptedClient(
[
[AssistantText(text="looks done to me", model="m")],
[finish("looks done to me", call_id="f1")],
[call("write_file", path="a.py", content="x")],
[AssistantText(text="now it is done", model="m")],
[finish("now it is done", call_id="f2")],
]
)
outcome = build(client, store, ws).run(milestone=1)
assert outcome.status == "completed"
assert (ws.root / "a.py").exists() # the nudge produced actual work
assert "not made any change yet" in client.calls[1]["system"]
assert (ws.root / "a.py").exists() # the refusal produced actual work
[refusal] = [
e.event
for e in store.read_events()
if isinstance(e.event, ToolResult) and e.event.call_id == "f1"
]
assert not refusal.success
assert "not made any change yet" in refusal.result
def test_persisting_in_an_empty_claim_escalates(
def test_persisting_in_an_empty_finish_escalates(
store: StateStore, ws: Workspace
) -> None:
prose = [AssistantText(text="nothing to do", model="m")]
client = ScriptedClient([prose, prose])
client = ScriptedClient([[finish(call_id="f1")], [finish(call_id="f2")]])
outcome = build(client, store, ws).run(milestone=1)
assert outcome.status == "escalate"
assert "without making any change" in outcome.note
def test_calls_after_finish_are_not_run(store: StateStore, ws: Workspace) -> None:
client = ScriptedClient(
[
[call("write_file", path="a.py", content="x")],
[finish(), call("write_file", call_id="late", path="b.py", content="y")],
]
)
assert build(client, store, ws).run(milestone=1).status == "completed"
assert not (ws.root / "b.py").exists()
[late] = [
e.event
for e in store.read_events()
if isinstance(e.event, ToolResult) and e.event.call_id == "late"
]
assert "finish had already ended the milestone" in late.result
def test_an_invented_tool_counts_toward_escalation(
store: StateStore, ws: Workspace
) -> None:
# Measured: a model on a server that doesn't enforce tool_choice will
# call tools that were never offered ("chat", "respond"). Without this it
# would burn every turn up to the cap.
client = ScriptedClient(
[[call("chat", call_id="x1")], [call("respond_user", call_id="x2")]]
)
outcome = build(client, store, ws, failure_limit=2).run(milestone=1)
assert outcome.status == "escalate"
assert len(client.calls) == 2
results = [e.event for e in store.read_events() if isinstance(e.event, ToolResult)]
assert "no such tool: chat" in results[0].result
assert "finish" in results[0].result # it is told what does exist
def test_two_consecutive_command_failures_escalate(
store: StateStore, ws: Workspace
) -> None:
@@ -150,7 +210,7 @@ def test_a_success_resets_the_failure_count(store: StateStore, ws: Workspace) ->
[call("run_command", command="exit 1")],
[call("run_command", command="true")],
[call("run_command", command="exit 1")],
[AssistantText(text="done", model="m")],
[finish()],
]
)
assert build(client, store, ws, failure_limit=2).run(milestone=1).status == (
@@ -168,7 +228,7 @@ def test_failed_reads_do_not_count_toward_escalation(
[call("read_file", path="missing-a.txt")],
[call("read_file", path="missing-b.txt")],
[call("write_file", path="a.py", content="x")],
[AssistantText(text="done", model="m")],
[finish()],
]
)
assert build(client, store, ws, failure_limit=2).run(milestone=1).status == (
@@ -182,8 +242,8 @@ def test_reading_alone_is_not_doing_the_work(store: StateStore, ws: Workspace) -
client = ScriptedClient(
[
[call("list_dir", path=".")],
[AssistantText(text="looks done to me", model="m")],
[AssistantText(text="really, done", model="m")],
[finish("looks done to me", call_id="f1")],
[finish("really, done", call_id="f2")],
]
)
outcome = build(client, store, ws).run(milestone=1)
@@ -214,7 +274,7 @@ def test_multiple_calls_in_one_turn_all_run(store: StateStore, ws: Workspace) ->
call("write_file", call_id="a", path="one.txt", content="1"),
call("write_file", call_id="b", path="two.txt", content="2"),
],
[AssistantText(text="done", model="m")],
[finish()],
]
)
build(client, store, ws).run(milestone=1)
@@ -222,4 +282,97 @@ def test_multiple_calls_in_one_turn_all_run(store: StateStore, ws: Workspace) ->
results = [
e.event for e in store.read_events() if type(e.event).__name__ == "ToolResult"
]
assert {r.call_id for r in results} == {"a", "b"}
assert {r.call_id for r in results} == {"a", "b", "fin"}
class StubDesk:
"""Stands in for ExpertDesk without a model behind it."""
def __init__(self, answer: str = "use sqlite", error: Exception | None = None):
self.asked: list[str] = []
self._answer = answer
self._error = error
def ask(self, question: str, **kw: Any) -> Any:
self.asked.append(question)
if self._error is not None:
raise self._error
class _A:
def render(self_inner) -> str:
return self._answer
return _A()
def test_ask_expert_is_offered_only_when_a_desk_exists(
store: StateStore, ws: Workspace
) -> None:
client = ScriptedClient([])
build(client, store, ws).run(milestone=1)
assert all(t["function"]["name"] != "ask_expert" for t in client.calls[0]["tools"])
client2 = ScriptedClient([])
build(client2, store, ws, expert=StubDesk()).run(milestone=1)
names = [t["function"]["name"] for t in client2.calls[0]["tools"]]
assert "ask_expert" in names
assert names[-1] == "finish"
# A12: in addition to the observable triggers, not instead of them.
assert "run_command" in names
def test_ask_expert_answer_comes_back_as_a_tool_result(
store: StateStore, ws: Workspace
) -> None:
desk = StubDesk(answer="prefer sqlite here")
client = ScriptedClient(
[
[call("ask_expert", question="sqlite or postgres?")],
[call("write_file", path="a.py", content="x")],
[finish()],
]
)
build(client, store, ws, expert=desk).run(milestone=1)
assert desk.asked == ["sqlite or postgres?"]
results = [
e.event for e in store.read_events() if type(e.event).__name__ == "ToolResult"
]
assert "prefer sqlite here" in results[0].result
def test_consulting_is_not_doing_the_work(store: StateStore, ws: Workspace) -> None:
# Asking is not changing anything, so a completion claim backed only by
# consults is still challenged.
client = ScriptedClient(
[
[call("ask_expert", question="what should I do?")],
[finish("I think it is done", call_id="f1")],
[finish("really done", call_id="f2")],
]
)
outcome = build(client, store, ws, expert=StubDesk()).run(milestone=1)
assert outcome.status == "escalate"
def test_exhausted_budget_is_reported_to_the_model_not_raised(
store: StateStore, ws: Workspace
) -> None:
from nisse_dev.expert import BudgetExhausted
desk = StubDesk(error=BudgetExhausted("consult budget of 3 is spent"))
client = ScriptedClient(
[
[call("ask_expert", question="again?")],
[call("write_file", path="a.py", content="x")],
[finish()],
]
)
outcome = build(client, store, ws, expert=desk).run(milestone=1)
assert outcome.status == "completed" # it got on with the work
results = [
e.event for e in store.read_events() if type(e.event).__name__ == "ToolResult"
]
assert not results[0].success
assert "budget" in results[0].result
nisse-dev/tests/test_expert.py
+186
View File
@@ -0,0 +1,186 @@
import io
from pathlib import Path
from typing import Any
import pytest
from nisse_dev.canonical import (
AssistantText,
CanonicalEvent,
DelegationCompleted,
DelegationStarted,
StoredEvent,
)
from nisse_dev.expert import (
DEFAULT_BUDGET,
BudgetExhausted,
ExpertAnswer,
ExpertDesk,
parse_answer,
)
from nisse_dev.llm import LLMError, RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
ROLES = RoleMap(
executor="llama3.2",
planner="haiku",
delegate="haiku",
consultant="claude-haiku-4-5",
)
GOOD = '{"answer": "Use SQLite; it is enough here.", "confidence": "high", "caveats": "Assumes one writer."}'
class ScriptedClient:
def __init__(self, replies: list[str]) -> None:
self._replies = replies
self.calls: list[dict[str, Any]] = []
def complete(
self,
events: list[StoredEvent],
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
) -> list[CanonicalEvent]:
self.calls.append(
{"events": events, "tools": tools, "model": model, "system": system}
)
if not self._replies:
return [AssistantText(text="nothing more", model=model)]
return [AssistantText(text=self._replies.pop(0), model=model)]
@pytest.fixture
def store(tmp_path: Path) -> StateStore:
return StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
def desk(client: Any, store: StateStore, **kw: Any) -> ExpertDesk:
return ExpertDesk(client, store, ROLES, **kw)
# --- parsing (the contract is prompt-level, so parsing is the guard) ------
def test_parses_a_clean_object() -> None:
parsed = parse_answer(GOOD)
assert parsed == ExpertAnswer(
answer="Use SQLite; it is enough here.",
confidence="high",
caveats="Assumes one writer.",
)
def test_parses_a_fenced_block_because_models_wrap_json() -> None:
assert parse_answer(f"Here you go:\n```json\n{GOOD}\n```") is not None
def test_missing_or_bad_fields_are_a_failed_turn() -> None:
assert parse_answer('{"answer": "", "confidence": "high"}') is None
assert parse_answer('{"answer": "x", "confidence": "certain"}') is None
assert parse_answer('{"confidence": "high"}') is None
assert parse_answer("no json at all") is None
assert parse_answer('["not", "an", "object"]') is None
def test_absent_caveats_is_tolerated() -> None:
parsed = parse_answer('{"answer": "yes", "confidence": "low"}')
assert parsed is not None
assert parsed.caveats == ""
# --- the desk ------------------------------------------------------------
def test_ask_returns_a_validated_answer(store: StateStore) -> None:
client = ScriptedClient([GOOD])
answer = desk(client, store).ask("SQLite or Postgres?")
assert answer.confidence == "high"
assert "SQLite" in answer.render()
def test_consult_is_stateless_and_uses_the_consultant_role(
store: StateStore,
) -> None:
store.append_event(AssistantText(text="secret repo context", model="m"))
client = ScriptedClient([GOOD])
desk(client, store).ask("Is this API deprecated?")
call = client.calls[0]
# A14: no repo access, no session state — exactly the question, and none
# of the log the caller was working from.
assert [e.event.text for e in call["events"]] == ["Is this API deprecated?"]
assert all("secret repo context" not in e.event.text for e in call["events"])
assert call["tools"] is None
assert call["model"] == "claude-haiku-4-5"
def test_unparseable_reply_is_retried_once_then_raises(store: StateStore) -> None:
client = ScriptedClient(["waffle", "more waffle"])
with pytest.raises(LLMError, match="no usable answer"):
desk(client, store).ask("q")
assert len(client.calls) == 2
assert "not a single valid JSON object" in client.calls[1]["system"]
def test_retry_succeeds_after_a_bad_first_reply(store: StateStore) -> None:
client = ScriptedClient(["waffle", GOOD])
assert desk(client, store).ask("q").confidence == "high"
def test_consults_are_recorded_as_composite_events(store: StateStore) -> None:
# A16 wants a consult/delegation eval set; reusing the pair gets consult
# evaluation for free.
desk(ScriptedClient([GOOD]), store).ask("q")
events = [e.event for e in store.read_events()]
started = next(e for e in events if isinstance(e, DelegationStarted))
completed = next(e for e in events if isinstance(e, DelegationCompleted))
assert started.role == "consultant"
assert started.brief == "q"
assert completed.exit_status == "answered"
def test_a_failed_consult_is_recorded_too(store: StateStore) -> None:
with pytest.raises(LLMError):
desk(ScriptedClient(["waffle", "waffle"]), store).ask("q")
completed = [
e.event for e in store.read_events() if isinstance(e.event, DelegationCompleted)
]
assert completed[0].exit_status == "failed"
def test_budget_is_enforced_across_calls(store: StateStore) -> None:
client = ScriptedClient([GOOD] * 10)
d = desk(client, store, budget=2)
d.ask("one")
d.ask("two")
with pytest.raises(BudgetExhausted, match="budget of 2"):
d.ask("three")
assert len(client.calls) == 2 # the third never reached the model
def test_budget_survives_a_restart(tmp_path: Path) -> None:
# Counted from the canonical log, not from memory, so a crash cannot
# hand a circling executor a fresh allowance.
db = tmp_path / "state.db"
with StateStore(db, oplog=OpLog("d", io.StringIO())) as store:
desk(ScriptedClient([GOOD]), store, budget=1).ask("one")
with (
StateStore(db, oplog=OpLog("d", io.StringIO())) as store,
pytest.raises(BudgetExhausted),
):
desk(ScriptedClient([GOOD]), store, budget=1).ask("two")
def test_budget_counts_only_after_the_task_marker(store: StateStore) -> None:
d = desk(ScriptedClient([GOOD] * 5), store, budget=1)
d.ask("earlier task")
marker = store.read_events()[-1].id
# A new task starts its allowance afresh from its own marker.
assert d.ask("new task", since_event_id=marker).confidence == "high"
def test_default_budget_is_small() -> None:
assert DEFAULT_BUDGET <= 5 # A14: do not let this become a second subsystem
nisse-dev/tests/test_gitea.py
-5
View File
@@ -162,11 +162,6 @@ def test_list_comments_passes_since() -> None:
assert seen["since"] == "2026-09-07T00:00:00Z"
def test_from_env_requires_config() -> None:
with pytest.raises(GiteaError, match="NISSE_GITEA_URL"):
GiteaClient.from_env({})
# --- reconciler (A3 external truth) --------------------------------------
nisse-dev/tests/test_llm.py
+244 -19
View File
@@ -1,5 +1,6 @@
import json
from pathlib import Path
from typing import Any
import httpx
import pytest
@@ -11,7 +12,17 @@ from nisse_dev.canonical import (
UserMessage,
)
from nisse_dev.envfile import load_env_file
from nisse_dev.llm import RATE_LIMIT_RETRY_S, LiteLLMClient, LLMError, RoleMap
from nisse_dev.llm import (
DEFAULT_MAX_OUTPUT_TOKENS,
RATE_LIMIT_RETRY_S,
AnthropicPassthroughClient,
BudgetExhausted,
LiteLLMClient,
LLMError,
RoleMap,
TruncatedResponse,
UpstreamUnavailable,
)
EVENTS = [StoredEvent(id=1, ts="t", event=UserMessage(text="hi", source="m:1"))]
@@ -131,6 +142,9 @@ def test_rate_limit_is_retried_once_then_surfaced() -> None:
client.complete(EVENTS, tools=None, model="local/qwen")
assert len(attempts) == 2
assert waited == [RATE_LIMIT_RETRY_S]
# Reported even though the call failed, so a caller can leave the wait
# out of how long the model took.
assert client.last_rate_limit_wait_s == RATE_LIMIT_RETRY_S
def test_rate_limit_that_clears_succeeds() -> None:
@@ -138,6 +152,19 @@ def test_rate_limit_that_clears_succeeds() -> None:
client = make_client(lambda request: responses.pop(0), sleep=lambda _s: None)
[event] = client.complete(EVENTS, tools=None, model="local/qwen")
assert isinstance(event, AssistantText)
assert client.last_rate_limit_wait_s == RATE_LIMIT_RETRY_S
def test_the_rate_limit_wait_is_per_call() -> None:
responses = [
httpx.Response(429, text="slow down"),
completion({"content": "one"}),
completion({"content": "two"}),
]
client = make_client(lambda request: responses.pop(0), sleep=lambda _s: None)
client.complete(EVENTS, tools=None, model="local/qwen")
client.complete(EVENTS, tools=None, model="local/qwen")
assert client.last_rate_limit_wait_s == 0.0
def test_connection_failure_raises_llm_error() -> None:
@@ -154,16 +181,16 @@ def test_malformed_response_raises_llm_error() -> None:
client.complete(EVENTS, tools=None, model="local/qwen")
ROLE_ENV = {
"NISSE_ROLE_EXECUTOR": "local/qwen",
"NISSE_ROLE_PLANNER": "anthropic/claude-opus-5",
"NISSE_ROLE_DELEGATE": "anthropic/claude-opus-5",
"NISSE_ROLE_CONSULTANT": "anthropic/claude-opus-5",
}
ROLES = RoleMap(
executor="local/qwen",
planner="anthropic/claude-opus-5",
delegate="anthropic/claude-opus-5",
consultant="anthropic/claude-opus-5",
)
def test_role_map_resolves_and_pins() -> None:
roles = RoleMap.from_env(ROLE_ENV)
roles = ROLES
assert roles.resolve("executor") == "local/qwen"
assert roles.pinned() == {
"executor": "local/qwen",
@@ -173,19 +200,9 @@ def test_role_map_resolves_and_pins() -> None:
}
def test_role_map_missing_env_raises() -> None:
with pytest.raises(LLMError, match="NISSE_ROLE_PLANNER"):
RoleMap.from_env({"NISSE_ROLE_EXECUTOR": "local/qwen"})
def test_unknown_role_raises() -> None:
with pytest.raises(LLMError, match="unknown role"):
RoleMap.from_env(ROLE_ENV).resolve("boss")
def test_client_from_env_requires_config() -> None:
with pytest.raises(LLMError, match="NISSE_LLM_BASE_URL"):
LiteLLMClient.from_env({})
ROLES.resolve("boss")
def test_load_env_file(tmp_path: Path) -> None:
@@ -203,3 +220,211 @@ def test_load_env_file_missing_is_noop(tmp_path: Path) -> None:
env: dict[str, str] = {}
load_env_file(tmp_path / "nope.env", env)
assert env == {}
# --- truncation (silent until measured against Opus) ---------------------
def test_openai_route_raises_on_truncation() -> None:
# finish_reason "length" means the answer was cut off; parsing it as
# complete yields a tool call with required arguments simply absent.
truncated = httpx.Response(
200,
json={
"model": "gpt-test-1",
"choices": [
{
"finish_reason": "length",
"message": {
"content": None,
"tool_calls": [
{
"id": "c",
"type": "function",
"function": {
"name": "submit_plan",
"arguments": '{"type": "arch"}',
},
}
],
},
}
],
},
)
client = make_client(lambda request: truncated)
with pytest.raises(TruncatedResponse, match="output limit"):
client.complete(EVENTS, tools=None, model="gpt-6-astra")
def test_anthropic_route_raises_on_truncation() -> None:
# Observed for real: Opus spent 4,064 output tokens against a 4,096
# ceiling, and a longer plan came back as submit_plan with no content.
def handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(
200,
json={
"model": "claude-opus-5",
"stop_reason": "max_tokens",
"content": [
{
"type": "tool_use",
"id": "t",
"name": "submit_plan",
"input": {"type": "arch"},
}
],
},
)
client = AnthropicPassthroughClient(
"https://litellm.test",
"tok",
transport=httpx.MockTransport(handler),
sleep=lambda _s: None,
)
with pytest.raises(TruncatedResponse, match="output limit"):
client.complete(EVENTS, tools=None, model="claude-opus-5")
def test_a_complete_answer_is_not_mistaken_for_truncation() -> None:
def handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(
200,
json={
"model": "claude-opus-5",
"stop_reason": "tool_use",
"content": [
{
"type": "tool_use",
"id": "t",
"name": "submit_plan",
"input": {"type": "arch", "content": "a real plan"},
}
],
},
)
client = AnthropicPassthroughClient(
"https://litellm.test", "tok", transport=httpx.MockTransport(handler)
)
[call] = client.complete(EVENTS, tools=None, model="claude-opus-5")
assert call.arguments["content"] == "a real plan"
def test_output_ceiling_is_generous_by_default() -> None:
# Output is billed by what is produced, so a low ceiling buys nothing and
# costs silent truncation. The configured value is tested in test_config.
assert DEFAULT_MAX_OUTPUT_TOKENS >= 8192
assert LiteLLMClient("http://x", "t")._max_tokens == DEFAULT_MAX_OUTPUT_TOKENS
def test_truncation_is_an_llm_error_so_callers_already_handle_it() -> None:
# The planner retries, the executor and delegation gate. None of them
# need to learn a new exception type.
assert issubclass(TruncatedResponse, LLMError)
def test_a_5xx_is_upstream_trouble() -> None:
client = make_client(lambda request: httpx.Response(503, text="bad gateway"))
with pytest.raises(UpstreamUnavailable, match="503"):
client.complete(EVENTS, tools=None, model="local-fast")
def test_a_connection_failure_is_upstream_trouble() -> None:
def handler(request: httpx.Request) -> httpx.Response:
raise httpx.ConnectError("refused")
with pytest.raises(UpstreamUnavailable, match="request failed"):
make_client(handler).complete(EVENTS, tools=None, model="local-fast")
def test_a_persistent_rate_limit_is_upstream_trouble() -> None:
# It cleared for neither attempt, so the caller should keep its work
# rather than discard it.
client = make_client(lambda request: httpx.Response(429, text="slow down"))
with pytest.raises(UpstreamUnavailable):
client.complete(EVENTS, tools=None, model="local-fast")
def test_a_bad_request_is_not_upstream_trouble() -> None:
client = make_client(lambda request: httpx.Response(400, text="bad schema"))
with pytest.raises(LLMError) as excinfo:
client.complete(EVENTS, tools=None, model="local-fast")
assert not isinstance(excinfo.value, UpstreamUnavailable)
def test_budget_refusal_is_a_kind_of_upstream_trouble() -> None:
# So one catch covers the family, but it can still be told apart where
# the difference matters — it will not clear within the hour.
assert issubclass(BudgetExhausted, UpstreamUnavailable)
# --- tool_choice (A15: constrained decoding) --------------------------------
TOOLS = [
{
"type": "function",
"function": {
"name": "finish",
"description": "done",
"parameters": {"type": "object", "properties": {}},
},
}
]
def capture_bodies() -> tuple[list[dict], Any]:
bodies: list[dict] = []
def handler(request: httpx.Request) -> httpx.Response:
bodies.append(json.loads(request.content))
return completion({"role": "assistant", "content": "ok"})
return bodies, handler
def test_required_is_sent_only_when_asked_and_tools_are_offered() -> None:
bodies, handler = capture_bodies()
client = make_client(handler)
client.complete(EVENTS, tools=TOOLS, model="m", require_tool_call=True)
client.complete(EVENTS, tools=TOOLS, model="m")
client.complete(EVENTS, tools=None, model="m", require_tool_call=True)
assert bodies[0]["tool_choice"] == "required"
assert "tool_choice" not in bodies[1] # the server's default, auto
assert "tool_choice" not in bodies[2] # nothing to choose from
def test_the_passthrough_asks_in_anthropics_terms() -> None:
bodies: list[dict] = []
def handler(request: httpx.Request) -> httpx.Response:
bodies.append(json.loads(request.content))
return httpx.Response(
200,
json={
"model": "claude-haiku-4-5",
"stop_reason": "tool_use",
"content": [
{"type": "tool_use", "id": "t", "name": "finish", "input": {}}
],
},
)
client = AnthropicPassthroughClient(
"https://litellm.test", "tok", transport=httpx.MockTransport(handler)
)
client.complete(
EVENTS, tools=TOOLS, model="claude-haiku-4-5", require_tool_call=True
)
assert bodies[0]["tool_choice"] == {"type": "any"}
def test_the_answering_model_is_recorded() -> None:
# A proxy fallback answers under another model's name; this is how a
# caller can tell.
client = make_client(
lambda request: completion({"role": "assistant", "content": "hi"})
)
client.complete(EVENTS, tools=None, model="local-fast")
assert client.last_model == "gpt-test-1"
nisse-dev/tests/test_matrix.py
-17
View File
@@ -143,20 +143,3 @@ def test_start_without_handler_raises() -> None:
)
with pytest.raises(TransportError, match="no message handler"):
transport.start()
def test_config_from_env_reports_missing() -> None:
with pytest.raises(TransportError, match="NISSE_MATRIX_TOKEN"):
MatrixConfig.from_env({"NISSE_MATRIX_HOMESERVER": "https://m"})
def test_config_from_env_custom_room_var() -> None:
env = {
"NISSE_MATRIX_HOMESERVER": "https://m/",
"NISSE_MATRIX_USER_ID": "@nisse-dev:test",
"NISSE_MATRIX_TOKEN": "t",
"NISSE_MATRIX_TEST_ROOM_ID": "!t:test",
}
config = MatrixConfig.from_env(env, room_var="NISSE_MATRIX_TEST_ROOM_ID")
assert config.room_id == "!t:test"
assert config.homeserver == "https://m" # trailing slash stripped
nisse-dev/tests/test_planner.py
+19 -1
View File
@@ -20,7 +20,7 @@ from nisse_dev.control import (
PlanningError,
SubmitPlan,
)
from nisse_dev.llm import RoleMap
from nisse_dev.llm import BudgetExhausted, RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.planner import PLANNER_TOOLS, LLMPlanningAgent
from nisse_dev.state import StateStore
@@ -279,3 +279,21 @@ def test_rejection_feedback_reaches_the_planner_via_the_log(
isinstance(e, UserMessage) and "too vague" in e.text for e in replan_context
)
assert "v2" in transport.outbound[-1].text
def test_budget_exhaustion_is_not_retried(store: StateStore) -> None:
"""A budget will not reappear within a turn, so a retry buys nothing and
delays telling the operator."""
class BrokeClient:
def __init__(self) -> None:
self.calls = 0
def complete(self, *args: Any, **kwargs: Any):
self.calls += 1
raise BudgetExhausted("daily budget exhausted")
client = BrokeClient()
with pytest.raises(BudgetExhausted):
LLMPlanningAgent(client, store, ROLES).plan("arch", "demo")
assert client.calls == 1 # not two
nisse-dev/tests/test_probe.py
+89
View File
@@ -0,0 +1,89 @@
import json
import httpx
from nisse_dev.probe import interpret, probe
def test_a_cut_at_the_declared_window_matches() -> None:
result = interpret(declared=4096, sent=6144, reported=4096, error=None)
assert result.verdict == "matches"
assert result.server_window == 4096
def test_a_cut_below_the_declaration_says_what_to_set() -> None:
# The dangerous case: prompts between the two would be silently cut.
result = interpret(declared=8192, sent=12288, reported=4096, error=None)
assert result.verdict == "declared_too_high"
assert "set max_input_tokens to 4096" in result.message
def test_a_cut_above_the_declaration_is_conservative() -> None:
result = interpret(declared=4096, sent=6144, reported=5000, error=None)
assert result.verdict == "declared_conservative"
def test_an_uncut_prompt_means_the_server_allows_more() -> None:
result = interpret(declared=4096, sent=6144, reported=6100, error=None)
assert result.verdict == "declared_conservative"
assert result.server_window is None
def test_a_refusal_is_the_safe_behaviour() -> None:
result = interpret(declared=4096, sent=6144, reported=None, error="400: too long")
assert result.verdict == "refuses"
def test_probe_calibrates_then_sends_half_again_the_window() -> None:
"""Against a fake Ollama: 2 tokens a word, silently cut at 4096."""
sizes: list[int] = []
def handler(request: httpx.Request) -> httpx.Response:
words = len(json.loads(request.content)["messages"][0]["content"].split())
sizes.append(words * 2)
reported = min(words * 2, 4096)
return httpx.Response(
200, json={"model": "llama3.2", "usage": {"prompt_tokens": reported}}
)
with httpx.Client(
base_url="http://litellm", transport=httpx.MockTransport(handler)
) as http:
result = probe(http, "llama3.2", declared=4096)
assert result.verdict == "matches"
assert sizes[1] >= 4096 * 1.4 # the probe really exceeded the window
def test_a_fallback_answer_is_never_credited_to_the_model() -> None:
"""Measured: LiteLLM answered an oversized prompt to local-coder with
claude-sonnet-5, and the probe called the refusing server roomy."""
def handler(request: httpx.Request) -> httpx.Response:
words = len(json.loads(request.content)["messages"][0]["content"].split())
model = "local-coder" if words < 1000 else "claude-sonnet-5"
return httpx.Response(
200, json={"model": model, "usage": {"prompt_tokens": words * 2}}
)
with httpx.Client(
base_url="http://litellm", transport=httpx.MockTransport(handler)
) as http:
result = probe(http, "local-coder", declared=32768, backing="openai/qwen3-14b")
assert result.verdict == "fallback"
assert "claude-sonnet-5 answered instead of local-coder" in result.message
def test_a_provider_reporting_its_own_model_name_is_not_a_fallback() -> None:
def handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(
200, json={"model": "qwen3-14b", "usage": {"prompt_tokens": 400}}
)
with httpx.Client(
base_url="http://litellm", transport=httpx.MockTransport(handler)
) as http:
result = probe(http, "local-coder", declared=32768, backing="openai/qwen3-14b")
assert result.verdict != "fallback"
nisse-dev/tests/test_stop.py
+356
View File
@@ -0,0 +1,356 @@
"""The operator's kill switch, `stop` (spec A4)."""
import io
import threading
from pathlib import Path
from typing import Any
from nisse_dev.cancel import STOPPED_RESULT, Cancel
from nisse_dev.canonical import AssistantText, ToolCall, ToolResult, UserMessage
from nisse_dev.control import (
AWAIT_ARCH_SIGNOFF,
AWAIT_INPUT,
AWAIT_MILESTONE_SIGNOFF,
Orchestrator,
PlanningStopped,
StubAgent,
SubmitPlan,
)
from nisse_dev.delegation import DelegationResult
from nisse_dev.executor import ExecutorOutcome, LLMExecutor
from nisse_dev.llm import RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.planner import LLMPlanningAgent
from nisse_dev.state import StateStore
from nisse_dev.tools import ToolOutcome
from nisse_dev.transport import InMemoryTransport
from nisse_dev.worker import SerialWorker
ROLES = RoleMap(
executor="llama3.2",
planner="claude-haiku-4-5",
delegate="claude-haiku-4-5",
consultant="claude-haiku-4-5",
)
def new_store(path: Path) -> StateStore:
return StateStore(path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
def user_messages(store: StateStore) -> list[str]:
return [
e.event.text for e in store.read_events() if isinstance(e.event, UserMessage)
]
# --- the orchestrator ------------------------------------------------------
class OperatorDuringWork:
"""An executor that hears from the operator mid-milestone, then does what
the real loop does at its next tool boundary."""
def __init__(self, cancel: Cancel, says: str) -> None:
self.cancel = cancel
self.says = says
self.transport: InMemoryTransport | None = None
def run(self, milestone: int) -> ExecutorOutcome:
assert self.transport is not None
self.transport.receive("don", self.says)
if self.cancel.requested:
return ExecutorOutcome("stopped", "during turn 3")
return ExecutorOutcome("completed", "done")
def at_start_of_milestone(
tmp_path: Path, says: str
) -> tuple[StateStore, InMemoryTransport, Cancel]:
store = new_store(tmp_path)
transport = InMemoryTransport()
cancel = Cancel()
executor = OperatorDuringWork(cancel, says)
executor.transport = transport
orch = Orchestrator(
store,
transport,
StubAgent(),
operators={"don"},
executor=executor,
cancel=cancel,
)
orch.start()
for text in ("use demo", "approve", "approve"):
transport.receive("don", text)
return store, transport, cancel
def test_stop_halts_a_running_milestone_and_asks_why(tmp_path: Path) -> None:
store, transport, cancel = at_start_of_milestone(tmp_path, "stop")
transport.receive("don", "start milestone 1")
assert store.get_state().state == AWAIT_INPUT
prompt = transport.outbound[-1]
assert prompt.kind == "question"
assert "stopped milestone 1 (during turn 3)" in prompt.text
assert "Why did you stop it?" in prompt.text
assert user_messages(store)[-1] == "stop"
assert not cancel.requested # the next piece of work is not pre-stopped
def test_the_reason_is_recorded_and_the_plan_stays_approved(tmp_path: Path) -> None:
store, transport, _ = at_start_of_milestone(tmp_path, "stop")
transport.receive("don", "start milestone 1")
transport.receive("don", "it was editing the wrong module")
assert store.get_state().state == AWAIT_MILESTONE_SIGNOFF
# The next run renders this, so the model hears why it was stopped.
assert user_messages(store)[-1] == "it was editing the wrong module"
assert "start milestone 1" in transport.outbound[-1].text
def test_stop_with_nothing_running_says_so_and_changes_nothing(
tmp_path: Path,
) -> None:
store = new_store(tmp_path)
transport = InMemoryTransport()
orch = Orchestrator(store, transport, StubAgent(), operators={"don"})
orch.start()
transport.receive("don", "use demo")
transport.receive("don", "stop")
# At a gate, "stop" is the command, not rejection feedback.
assert store.get_state().state == AWAIT_ARCH_SIGNOFF
assert orch._data["revs"]["arch"] == 1
assert (
transport.outbound[-1].text == f"nothing to stop (state={AWAIT_ARCH_SIGNOFF})"
)
def test_stop_from_a_non_operator_is_ignored(tmp_path: Path) -> None:
_, transport, cancel = at_start_of_milestone(tmp_path, "approve")
before = len(transport.outbound)
transport.receive("mallory", "stop")
assert len(transport.outbound) == before
assert not cancel.requested
def test_other_messages_during_work_are_answered_not_acted_on(
tmp_path: Path,
) -> None:
store, transport, _ = at_start_of_milestone(tmp_path, "approve")
transport.receive("don", "start milestone 1")
texts = [m.text for m in transport.outbound]
assert "busy (EXECUTING); say 'stop' to halt" in texts
# Recorded, as every operator message is, but not applied to anything.
assert user_messages(store)[-1] == "approve"
class StopsBeforePlanning:
def __init__(self, cancel: Cancel) -> None:
self.cancel = cancel
self.transport: InMemoryTransport | None = None
self.answers: list[str | None] = []
def plan(
self,
gate: str,
project: str,
feedback: str | None = None,
answer: str | None = None,
) -> SubmitPlan:
self.answers.append(answer)
if len(self.answers) == 1:
assert self.transport is not None
self.transport.receive("don", "stop")
if self.cancel.requested:
raise PlanningStopped("stopped by the operator")
return SubmitPlan(f"{gate} plan, after: {answer}")
def test_stop_during_planning_resumes_planning_with_the_reason(
tmp_path: Path,
) -> None:
store = new_store(tmp_path)
transport = InMemoryTransport()
cancel = Cancel()
planner = StopsBeforePlanning(cancel)
planner.transport = transport
orch = Orchestrator(store, transport, planner, operators={"don"}, cancel=cancel)
orch.start()
transport.receive("don", "use demo")
assert store.get_state().state == AWAIT_INPUT
assert "stopped planning arch" in transport.outbound[-1].text
transport.receive("don", "keep it to one module")
assert store.get_state().state == AWAIT_ARCH_SIGNOFF
assert planner.answers[-1] == "keep it to one module"
# --- on real threads -------------------------------------------------------
class BlocksUntilStopped:
def __init__(self, cancel: Cancel) -> None:
self.cancel = cancel
self.running = threading.Event()
def run(self, milestone: int) -> ExecutorOutcome:
self.running.set()
for _ in range(500):
if self.cancel.requested:
return ExecutorOutcome("stopped", "at a tool boundary")
threading.Event().wait(0.01)
return ExecutorOutcome("completed", "never stopped")
def test_stop_reaches_work_running_on_the_worker(tmp_path: Path) -> None:
# The bug this design fixes: the work used to run on the transport's own
# thread, so no message, stop included, was read until it finished.
worker = SerialWorker()
worker.start()
store = worker.call(lambda: new_store(tmp_path))
transport = InMemoryTransport()
cancel = Cancel()
executor = BlocksUntilStopped(cancel)
orch = Orchestrator(
store,
transport,
StubAgent(),
operators={"don"},
executor=executor,
worker=worker,
cancel=cancel,
)
worker.call(orch.start)
for text in ("use demo", "approve", "approve", "start milestone 1"):
transport.receive("don", text)
assert executor.running.wait(5)
transport.receive("don", "status") # delivered while the milestone runs
transport.receive("don", "stop")
worker.call(lambda: None) # wait for the stop to be recorded
texts = [m.text for m in transport.outbound]
assert "busy (EXECUTING); say 'stop' to halt" in texts
assert worker.call(store.get_state).state == AWAIT_INPUT
assert "stopped milestone 1 (at a tool boundary)" in texts[-1]
worker.call(store.close)
worker.stop(timeout=5)
# --- the loops -------------------------------------------------------------
class ScriptedClient:
def __init__(self, turns: list[list[Any]], on_call: Any = None) -> None:
self.turns = turns
self.on_call = on_call
self.calls = 0
def complete(self, events: Any, tools: Any, model: str, system: Any = None):
self.calls += 1
if self.on_call:
self.on_call()
return self.turns.pop(0) if self.turns else [AssistantText("done", model)]
class StopsAfterFirstCall:
"""A workspace whose first tool call is where the operator says stop."""
def __init__(self, cancel: Cancel) -> None:
self.cancel = cancel
self.ran: list[str] = []
def describe(self) -> str:
return "workspace"
def dispatch(self, tool: str, arguments: dict[str, Any]) -> ToolOutcome:
self.ran.append(tool)
self.cancel.set()
return ToolOutcome("ok", success=True)
def test_executor_stops_before_a_turn_without_calling_the_model(
tmp_path: Path,
) -> None:
cancel = Cancel()
cancel.set()
client = ScriptedClient([])
executor = LLMExecutor(
client, new_store(tmp_path), ROLES, StopsAfterFirstCall(cancel), cancel=cancel
)
outcome = executor.run(1)
assert outcome == ExecutorOutcome("stopped", "before turn 1")
assert client.calls == 0
def test_executor_stops_between_tool_calls_and_records_the_skipped_ones(
tmp_path: Path,
) -> None:
cancel = Cancel()
store = new_store(tmp_path)
ws = StopsAfterFirstCall(cancel)
calls = [
ToolCall(call_id="a", tool="run_command", arguments={"command": "make"}),
ToolCall(call_id="b", tool="write_file", arguments={"path": "x"}),
]
executor = LLMExecutor(ScriptedClient([calls]), store, ROLES, ws, cancel=cancel)
outcome = executor.run(1)
assert outcome == ExecutorOutcome("stopped", "during turn 1")
assert ws.ran == ["run_command"]
results = {
e.event.call_id: e.event.result
for e in store.read_events()
if isinstance(e.event, ToolResult)
}
# Every call has a result, so the transcript still renders.
assert results == {"a": "ok", "b": STOPPED_RESULT}
class StoppedDelegate:
def run(self, *, brief: str, branch: str) -> DelegationResult:
return DelegationResult("stopped", "stopped by the operator during turn 2")
def test_a_stopped_delegation_stops_the_milestone(tmp_path: Path) -> None:
failing = [
[ToolCall(call_id=f"c{n}", tool="run_command", arguments={"command": "x"})]
for n in range(2)
]
class Failing:
def describe(self) -> str:
return "workspace"
def dispatch(self, tool: str, arguments: dict[str, Any]) -> ToolOutcome:
return ToolOutcome("exit 1", success=False)
executor = LLMExecutor(
ScriptedClient(failing),
new_store(tmp_path),
ROLES,
Failing(),
delegate=StoppedDelegate(),
)
outcome = executor.run(1)
assert outcome.status == "stopped"
assert "during a delegation" in outcome.note
def test_planner_does_not_start_a_turn_once_stopped(tmp_path: Path) -> None:
cancel = Cancel()
cancel.set()
client = ScriptedClient([])
planner = LLMPlanningAgent(client, new_store(tmp_path), ROLES, cancel=cancel)
try:
planner.plan("arch", "demo")
except PlanningStopped:
pass
else:
raise AssertionError("planning ran after stop")
assert client.calls == 0
nisse-dev/tests/test_tools.py
+1 -1
View File
@@ -126,7 +126,7 @@ def test_stdin_is_closed_so_prompts_fail_fast(ws: Workspace) -> None:
def test_long_output_is_truncated(ws: Workspace) -> None:
outcome = ws.run_command("printf 'x%.0s' $(seq 1 20000)")
assert len(outcome.result) < MAX_OUTPUT_CHARS + 200
assert "truncated" in outcome.result
assert "characters omitted" in outcome.result
def test_dispatch_unknown_tool_and_missing_argument(ws: Workspace) -> None:
nisse-dev/tests/test_worker.py
+83
View File
@@ -0,0 +1,83 @@
import io
import threading
from pathlib import Path
import pytest
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
from nisse_dev.worker import InlineWorker, SerialWorker
@pytest.fixture
def worker() -> SerialWorker:
w = SerialWorker()
w.start()
yield w
w.stop(timeout=5)
def test_jobs_run_in_order_on_the_worker_thread(worker: SerialWorker) -> None:
seen: list[tuple[int, str]] = []
for n in range(3):
worker.submit(lambda n=n: seen.append((n, threading.current_thread().name)))
worker.call(lambda: None) # drains everything queued before it
assert [n for n, _ in seen] == [0, 1, 2]
assert {name for _, name in seen} == {"nisse-worker"}
def test_call_returns_the_result_and_raises_the_error(worker: SerialWorker) -> None:
assert worker.call(lambda: 42) == 42
def boom() -> None:
raise ValueError("bad job")
with pytest.raises(ValueError, match="bad job"):
worker.call(boom)
def test_a_failed_job_is_reported_and_the_worker_carries_on() -> None:
errors: list[BaseException] = []
worker = SerialWorker(on_error=errors.append)
worker.start()
def boom() -> None:
raise RuntimeError("lost job")
worker.submit(boom)
assert worker.call(lambda: "still here") == "still here"
assert [str(e) for e in errors] == ["lost job"]
worker.stop(timeout=5)
def test_call_from_the_worker_itself_runs_inline(worker: SerialWorker) -> None:
# Queueing behind itself would deadlock.
assert worker.call(lambda: worker.call(lambda: "nested")) == "nested"
def test_the_store_is_usable_only_from_the_thread_that_owns_it(
worker: SerialWorker, tmp_path: Path
) -> None:
store = worker.call(
lambda: StateStore(tmp_path / "state.db", oplog=OpLog("t", io.StringIO()))
)
assert worker.call(store.get_state) is None
# The tripwire the worker design keeps on: any other thread fails loudly.
with pytest.raises(Exception, match="same thread"):
store.get_state()
worker.call(store.close)
def test_inline_worker_defers_a_job_submitted_from_inside_a_job() -> None:
worker = InlineWorker()
order: list[str] = []
def outer() -> None:
order.append("outer start")
worker.submit(lambda: order.append("inner"))
order.append("outer end")
worker.submit(outer)
assert order == ["outer start", "outer end", "inner"]
assert not worker.busy
scripts/benchmark.py
+124
View File
@@ -0,0 +1,124 @@
"""Benchmark executor models on canned milestones, and compare the results.
uv run python scripts/benchmark.py run --models A,B [--runs 5] [--tasks T,...]
[--note "vLLM 0.29, AWQ, fp8 KV"] [--config PATH]
uv run python scripts/benchmark.py report [--models A,B] [--tasks T,...]
`run` appends one JSON line per run to bench/results/, which is committed:
the report compares every run ever stored, so benchmarking a new model means
running only that model. See docs/Tuning-Guide.md, "Benchmarking models".
"""
import argparse
import os
import sys
from pathlib import Path
from nisse_dev.agent import make_llm_clients
from nisse_dev.bench.results import append, load, render, results_file, summarise
from nisse_dev.bench.runner import agent_commit, run_task
from nisse_dev.bench.tasks import load_tasks
from nisse_dev.catalog import ModelCatalog
from nisse_dev.config import AgentConfig, ConfigError
from nisse_dev.envfile import load_env_file
REPO = Path(__file__).resolve().parents[1]
RESULTS = REPO / "bench" / "results"
def _names(raw: str | None) -> list[str] | None:
return [n.strip() for n in raw.split(",") if n.strip()] if raw else None
def run(args: argparse.Namespace) -> int:
load_env_file(Path(os.environ.get("NISSE_ENV", ".env")).expanduser())
try:
config = (
AgentConfig.from_file(Path(args.config))
if args.config
else AgentConfig.load()
)
except ConfigError as exc:
print(exc, file=sys.stderr)
return 2
models = _names(args.models) or []
tasks = load_tasks(_names(args.tasks))
catalog = ModelCatalog.discover(config.llm_base_url, config.llm_token)
catalog.require_limits({m: m for m in models})
normalized, _passthrough = make_llm_clients(config, catalog)
commit = agent_commit(REPO)
for model in models:
facts = catalog.facts(model)
out = results_file(args.results, model)
for task in tasks:
for n in range(args.runs):
# The executor always takes the normalized route in the agent,
# so it does here too.
record = run_task(
task,
facts,
normalized,
config.tuning,
note=args.note,
commit=commit,
)
append(out, record)
verdict = "pass" if record.success else f"FAIL ({record.outcome})"
extra = (
f" fallback to {record.fallbacks[0]}!" if record.fallbacks else ""
)
print(
f"{model} {task.name} run {n + 1}/{args.runs}: {verdict}, "
f"{record.turns} turns, {record.wall_s:.0f}s, "
f"${record.cost:.4f}{extra}",
flush=True,
)
print()
return report(args)
def report(args: argparse.Namespace) -> int:
records = load(args.results)
models = _names(getattr(args, "models", None))
tasks = _names(getattr(args, "tasks", None))
if models:
records = [r for r in records if r["model"] in models or r["backing"] in models]
if tasks:
records = [r for r in records if r["task"] in tasks]
if not records:
print("no stored results match", file=sys.stderr)
return 1
current = {task.name: task.content_hash for task in load_tasks()}
print(render(summarise(records, current)))
return 0
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--results", type=Path, default=RESULTS, help=argparse.SUPPRESS)
sub = parser.add_subparsers(dest="command", required=True)
run_p = sub.add_parser("run", help="benchmark models, then report")
run_p.add_argument(
"--models", required=True, help="LiteLLM aliases, comma separated"
)
run_p.add_argument("--runs", type=int, default=5, help="runs per task (default 5)")
run_p.add_argument("--tasks", help="task names, comma separated (default: all)")
run_p.add_argument(
"--note", default="", help='how the model was served, e.g. "vLLM 0.29, AWQ"'
)
run_p.add_argument("--config", help="agent config (default: NISSE_CONFIG)")
run_p.set_defaults(handler=run)
report_p = sub.add_parser("report", help="compare stored results")
report_p.add_argument("--models", help="aliases or backing models to include")
report_p.add_argument("--tasks", help="tasks to include")
report_p.set_defaults(handler=report)
args = parser.parse_args()
return args.handler(args)
if __name__ == "__main__":
raise SystemExit(main())
scripts/evaluate_model.sh
Executable
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Evaluate a model for the executor role, from a declared LiteLLM alias to a
# benchmark report next to every model measured before it.
#
# scripts/evaluate_model.sh ALIAS "how it is served" [RUNS] [--config PATH]
#
# 1. The context probe: does the server's window match what LiteLLM declares?
# 2. The capability matrix: does it hold the tool contract, and is its
# supports_tool_choice declaration true?
# 3. The benchmark: canned milestones, RUNS times each (default 5).
# Stops at the first step that fails; there is no point benchmarking a model
# that can't hold the tool contract.
set -euo pipefail
cd "$(dirname "$0")/.."
alias="${1:?usage: evaluate_model.sh ALIAS NOTE [RUNS] [--config PATH]}"
note="${2:?give a note on how the model is served, e.g. \"vLLM 0.29, AWQ\"}"
runs="${3:-5}"
shift $(( $# < 3 ? $# : 3 ))
config=("$@")
echo "== 1/3 context probe"
uv run python scripts/probe_context.py "$alias" "${config[@]}"
echo "== 2/3 capability matrix"
uv run pytest -m integration -q -rs tests/integration/test_model_capability.py \
--capability-models="$alias"
echo "== 3/3 benchmark ($runs runs per task)"
uv run python scripts/benchmark.py run --models "$alias" --runs "$runs" \
--note "$note" "${config[@]}"
scripts/probe_context.py
+75
View File
@@ -0,0 +1,75 @@
"""Check a model's declared context window against what its server does.
uv run python scripts/probe_context.py ALIAS [--config PATH]
ALIAS is a LiteLLM model alias. The LiteLLM URL and token come from the agent
config (NISSE_CONFIG, ./nisse-dev.yaml, or --config), with its secrets from
the environment or .env. See docs/Tuning-Guide.md, "Declaring a local model's
limits", and nisse_dev/probe.py for what the verdicts mean.
"""
import argparse
import os
import sys
from pathlib import Path
import httpx
from nisse_dev.catalog import ModelCatalog
from nisse_dev.config import AgentConfig, ConfigError
from nisse_dev.envfile import load_env_file
from nisse_dev.probe import probe
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("alias", help="LiteLLM model alias to probe")
parser.add_argument("--config", help="agent config file to take LiteLLM from")
parser.add_argument(
"--max-probe-tokens",
type=int,
default=64_000,
help="refuse to send a prompt larger than this (default: %(default)s)",
)
args = parser.parse_args()
load_env_file(Path(os.environ.get("NISSE_ENV", ".env")).expanduser())
try:
config = (
AgentConfig.from_file(Path(args.config))
if args.config
else AgentConfig.load()
)
except ConfigError as exc:
print(exc, file=sys.stderr)
return 2
catalog = ModelCatalog.discover(config.llm_base_url, config.llm_token)
facts = catalog.facts(args.alias)
declared = facts.max_input_tokens
if declared is None:
print(
f"LiteLLM declares no max_input_tokens for {args.alias}; set one first",
file=sys.stderr,
)
return 1
if declared * 1.5 > args.max_probe_tokens:
print(
f"{args.alias} declares {declared} tokens; probing would send about "
f"{int(declared * 1.5)}. Raise --max-probe-tokens to allow it.",
file=sys.stderr,
)
return 1
with httpx.Client(
base_url=config.llm_base_url.rstrip("/"),
headers={"Authorization": f"Bearer {config.llm_token}"},
timeout=600,
) as http:
result = probe(http, args.alias, declared, facts.backing)
print(f"{args.alias}: {result.message}")
return 0 if result.verdict in ("matches", "refuses", "declared_conservative") else 1
if __name__ == "__main__":
raise SystemExit(main())
tests/conftest.py
+20
View File
@@ -0,0 +1,20 @@
"""Command-line options for the repository-level test suites.
Declared here rather than beside the tests that use them: pytest only reads
options from conftest files it loads at startup, and this directory is one
of the configured test paths.
"""
import pytest
def pytest_addoption(parser: pytest.Parser) -> None:
parser.addoption(
"--capability-models",
default="",
help=(
"comma-separated LiteLLM aliases to measure with the capability "
"matrix (tests/integration/test_model_capability.py). Spends real "
"tokens on every model listed; the matrix skips when this is empty."
),
)
tests/integration/conftest.py
+62
View File
@@ -0,0 +1,62 @@
"""Shared setup for the live integration tests.
They run against real services, configured by nisse-test.yaml beside this
file: an ordinary agent config aimed at test resources, committed to the
repository. Its secrets are ${…} references, supplied by CI as environment
variables and locally by .env at the repository root.
A missing secret is a failure, not a skip. A half-configured environment must
not pass quietly as a skip: that is how a broken CI credential hides.
"""
from pathlib import Path
import pytest
from nisse_dev.config import AgentConfig
from nisse_dev.envfile import load_env_file
from nisse_dev.gitea import GiteaClient
from nisse_dev.llm import AnthropicPassthroughClient, LiteLLMClient
REPO = Path(__file__).resolve().parents[2]
TEST_CONFIG = Path(__file__).parent / "nisse-test.yaml"
# The secrets nisse-test.yaml refers to, for local runs. CI sets them as real
# environment variables, which win over the file.
load_env_file(REPO / ".env")
@pytest.fixture(scope="session")
def live_config() -> AgentConfig:
return AgentConfig.from_file(TEST_CONFIG)
@pytest.fixture
def live_model(live_config: AgentConfig) -> str:
"""The model the live tests plan, execute and delegate with."""
return live_config.roles.planner
@pytest.fixture
def litellm(live_config: AgentConfig) -> LiteLLMClient:
return LiteLLMClient(
live_config.llm_base_url,
live_config.llm_token,
max_tokens=live_config.max_output_tokens,
)
@pytest.fixture
def passthrough(live_config: AgentConfig) -> AnthropicPassthroughClient:
return AnthropicPassthroughClient(
live_config.llm_base_url,
live_config.llm_token,
max_tokens=live_config.max_output_tokens,
)
@pytest.fixture
def gitea(live_config: AgentConfig) -> GiteaClient:
client = GiteaClient(live_config.gitea_url, live_config.gitea_token)
yield client
client.close()
tests/integration/nisse-test.yaml
+40
View File
@@ -0,0 +1,40 @@
# Agent config for the live integration tests, aimed at test resources: the
# testing identities, the dedicated test room, and the sandbox repo. Never
# point it at production.
#
# Secrets are ${…} references. CI passes them as environment variables;
# locally they come from .env at the repository root. A missing one fails
# the live tests that need it, rather than skipping them, so a broken
# credential cannot hide as a skip.
# Unused by the live tests, which work in pytest's temporary directories;
# present because every agent config has them.
state_db: /tmp/nisse-test/state.db
workspace: /tmp/nisse-test/work
project: nisse-dev-testing
llm:
base_url: https://litellm.dongar.net
token: ${NISSE_LLM_TOKEN}
# The live tests exercise the gated loop and the routes over a real proxy,
# not local-model competence (measured separately by test_model_capability),
# so every role uses a model that holds the tool-call contract.
roles:
executor: claude-haiku-4-5
planner: claude-haiku-4-5
delegate: claude-haiku-4-5
consultant: claude-haiku-4-5
matrix:
homeserver: https://matrix.dongar.net
user_id: "@nisse-dev-testing:dongar.net"
token: ${NISSE_MATRIX_TOKEN}
room_id: "!UxpXhTlQsFIeNZwkgv:dongar.net" # the test room, never production
operators: ["@dongar:dongar.net"]
gitea:
url: https://gitea.dongar.net
token: ${NISSE_GITEA_TOKEN}
repo: dongar/nisse-dev-testing-sandbox # the sandbox, never a real project
operators: [dongar]
tests/integration/test_benchmark_live.py
+31
View File
@@ -0,0 +1,31 @@
"""One live benchmark run: the simplest task, through the agent's own
client wiring, with the test config's executor."""
from pathlib import Path
import pytest
from nisse_dev.agent import make_llm_clients
from nisse_dev.bench.runner import run_task
from nisse_dev.bench.tasks import load_tasks
from nisse_dev.catalog import ModelCatalog
from nisse_dev.config import AgentConfig
pytestmark = pytest.mark.integration
def test_the_simplest_task_passes_live(
live_config: AgentConfig, tmp_path: Path
) -> None:
catalog = ModelCatalog.discover(live_config.llm_base_url, live_config.llm_token)
normalized, _ = make_llm_clients(live_config, catalog)
facts = catalog.facts(live_config.roles.executor)
[task] = load_tasks(["write-function"])
record = run_task(task, facts, normalized, live_config.tuning, workdir=tmp_path)
assert record.fallbacks == []
assert record.success, (
f"{record.outcome}: {record.outcome_note}\n{record.check_output}"
)
assert record.cost >= 0
tests/integration/test_delegation_live.py
+18 -20
View File
@@ -2,30 +2,18 @@
working code and a structured handback out."""
import io
import os
from pathlib import Path
import pytest
from nisse_dev.delegation import Caps, Delegation
from nisse_dev.envfile import load_env_file
from nisse_dev.git import GitRepo
from nisse_dev.llm import AnthropicPassthroughClient
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
from nisse_dev.tools import Workspace
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_LIVE_PLANNER_MODEL"
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no live frontier route configured ({SENTINEL} unset)",
),
]
pytestmark = pytest.mark.integration
BRIEF = """Create `slugify.py` in the workspace root containing a function
`slugify(text)` that lowercases the text and replaces every run of
@@ -34,7 +22,9 @@ hyphen. Then create `test_slugify.py` with at least two assertions covering
it, and run `python -m pytest test_slugify.py -q` until it passes."""
def build(tmp_path: Path, caps: Caps) -> tuple[Delegation, GitRepo, StateStore]:
def build(
tmp_path: Path, caps: Caps, client: AnthropicPassthroughClient, model: str
) -> tuple[Delegation, GitRepo, StateStore]:
repo = GitRepo(tmp_path / "work")
repo.init()
repo.configure_identity("nisse-dev-testing", "n@t.test")
@@ -42,18 +32,22 @@ def build(tmp_path: Path, caps: Caps) -> tuple[Delegation, GitRepo, StateStore]:
repo.commit_all("seed")
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
delegation = Delegation(
AnthropicPassthroughClient.from_env(),
client,
store,
Workspace(repo.root, timeout_s=60),
repo,
model=os.environ[SENTINEL],
model=model,
caps=caps,
)
return delegation, repo, store
def test_delegate_completes_a_scoped_subtask(tmp_path: Path) -> None:
delegation, repo, _store = build(tmp_path, Caps(turns=12, wall_clock_s=300.0))
def test_delegate_completes_a_scoped_subtask(
tmp_path: Path, passthrough: AnthropicPassthroughClient, live_model: str
) -> None:
delegation, repo, _store = build(
tmp_path, Caps(turns=12, wall_clock_s=300.0), passthrough, live_model
)
result = delegation.run(brief=BRIEF, branch="live/delegation-1")
@@ -76,10 +70,14 @@ def test_delegate_completes_a_scoped_subtask(tmp_path: Path) -> None:
assert check.success, f"delegate's own tests do not pass: {check.result[:400]}"
def test_a_cap_breach_discards_the_work(tmp_path: Path) -> None:
def test_a_cap_breach_discards_the_work(
tmp_path: Path, passthrough: AnthropicPassthroughClient, live_model: str
) -> None:
# One turn cannot finish the brief, so this exercises the abort path
# against a real model rather than a scripted one.
delegation, repo, _ = build(tmp_path, Caps(turns=1, wall_clock_s=120.0))
delegation, repo, _ = build(
tmp_path, Caps(turns=1, wall_clock_s=120.0), passthrough, live_model
)
result = delegation.run(brief=BRIEF, branch="live/delegation-2")
tests/integration/test_entry_points.py
+35 -4
View File
@@ -1,16 +1,47 @@
"""Integration tests: run the installed console scripts as real subprocesses."""
import os
import subprocess
from pathlib import Path
import pytest
pytestmark = pytest.mark.integration
@pytest.mark.parametrize("agent", ["nisse-dev", "nisse-ops"])
def test_entry_point_runs_and_greets(agent: str) -> None:
def test_ops_entry_point_runs_and_greets() -> None:
result = subprocess.run(
[agent], capture_output=True, text=True, timeout=30, check=False
["nisse-ops"], capture_output=True, text=True, timeout=30, check=False
)
assert result.returncode == 0
assert result.stdout == f"Hello from {agent}, the house spirit is awake.\n"
assert result.stdout == "Hello from nisse-ops, the house spirit is awake.\n"
def test_dev_entry_point_refuses_to_start_unconfigured(tmp_path: Path) -> None:
"""An incomplete config must fail at startup, naming everything absent.
Run from an empty directory, with an empty config file and a .env path
that does not exist: the repository's .env holds live credentials, and a
test must never be one cwd away from starting a real agent.
"""
config = tmp_path / "nisse-dev.yaml"
config.write_text("")
result = subprocess.run(
["nisse-dev"],
capture_output=True,
text=True,
timeout=30,
check=False,
cwd=tmp_path,
env={
"PATH": os.environ["PATH"],
"HOME": os.environ["HOME"],
"NISSE_ENV": "/nonexistent/.env",
"NISSE_CONFIG": str(config),
},
)
assert result.returncode == 2
assert "cannot start" in result.stderr
# All of them at once, not the first one it happened to reach.
for expected in ("state_db", "matrix.token", "gitea.repo"):
assert expected in result.stderr
tests/integration/test_expert_live.py
+54
View File
@@ -0,0 +1,54 @@
"""A real consult over the normalized route (A14).
The point is the prompt-level contract: a live model must produce something
that parses against our local schema, since no API-level enforcement exists.
"""
import io
from pathlib import Path
import pytest
from nisse_dev.evals import collect
from nisse_dev.expert import BudgetExhausted, ExpertDesk
from nisse_dev.llm import LiteLLMClient, RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
pytestmark = pytest.mark.integration
@pytest.fixture
def desk(tmp_path: Path, litellm: LiteLLMClient, live_model: str):
def make(budget: int = 2) -> tuple[ExpertDesk, StateStore]:
model = live_model
roles = RoleMap(executor=model, planner=model, delegate=model, consultant=model)
store = StateStore(
tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO())
)
return ExpertDesk(litellm, store, roles, budget=budget), store
return make
def test_live_consult_parses_against_the_local_schema(desk) -> None:
expert, store = desk()
answer = expert.ask(
"For a single-process agent storing a few thousand rows of task state, "
"is SQLite sufficient, or is PostgreSQL warranted? Answer briefly."
)
assert answer.confidence in {"high", "medium", "low"}
assert len(answer.answer) > 20
# The consult is captured for A16 without any extra bookkeeping.
[record] = collect(store)
assert record.role == "consultant"
assert record.exit_status == "answered"
def test_live_budget_stops_a_circling_caller(desk) -> None:
expert, _ = desk(budget=1)
expert.ask("Name one advantage of WAL mode in SQLite. Be brief.")
with pytest.raises(BudgetExhausted):
expert.ask("And another?")
tests/integration/test_gitea_live.py
+9 -24
View File
@@ -5,29 +5,17 @@ Cleans up after itself (closes the PR, deletes the branch) so repeated runs
do not silt up the sandbox.
"""
import os
import uuid
from pathlib import Path
import pytest
from nisse_dev.envfile import load_env_file
from nisse_dev.config import AgentConfig
from nisse_dev.git import GitError, GitRepo, authenticated_url
from nisse_dev.gitea import GiteaClient, pull_request_reconciler
from nisse_dev.state import Intent
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_GITEA_URL"
ALSO_REQUIRED = ("NISSE_GITEA_TOKEN", "NISSE_GITEA_TEST_REPO")
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no live Gitea configured ({SENTINEL} unset)",
),
]
pytestmark = pytest.mark.integration
def intent_for(branch: str) -> Intent:
@@ -41,16 +29,13 @@ def intent_for(branch: str) -> Intent:
)
def test_push_open_pr_and_reconcile(tmp_path: Path) -> None:
missing = [var for var in ALSO_REQUIRED if not os.environ.get(var)]
assert not missing, (
f"{SENTINEL} is set, so the live Gitea path is expected to work, "
f"but these are missing: {', '.join(missing)}"
)
base_url = os.environ[SENTINEL]
token = os.environ["NISSE_GITEA_TOKEN"]
repo_path = os.environ["NISSE_GITEA_TEST_REPO"]
client = GiteaClient.from_env()
def test_push_open_pr_and_reconcile(
tmp_path: Path, live_config: AgentConfig, gitea: GiteaClient
) -> None:
base_url = live_config.gitea_url
token = live_config.gitea_token
repo_path = live_config.gitea_repo # the sandbox
client = gitea
account = client.whoami()
assert "testing" in account, (
tests/integration/test_llm_live.py
+41 -33
View File
@@ -1,41 +1,19 @@
"""Live smoke test against the real LiteLLM proxy.
NISSE_LLM_BASE_URL alone decides whether this runs: if it is set, the live
path is expected to work, and anything else missing is a *failure*, not a
skip. A half-configured environment must not pass quietly as a skip — that is
how a broken CI credential hides.
"""
import os
from pathlib import Path
"""Live smoke test against the real LiteLLM proxy."""
import pytest
from nisse_dev.canonical import AssistantText, StoredEvent, UserMessage
from nisse_dev.envfile import load_env_file
from nisse_dev.catalog import ModelCatalog
from nisse_dev.config import AgentConfig
from nisse_dev.context import BudgetedClient
from nisse_dev.llm import LiteLLMClient
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_LLM_BASE_URL"
ALSO_REQUIRED = ("NISSE_LLM_TOKEN", "NISSE_ROLE_EXECUTOR")
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no live LLM configured ({SENTINEL} unset)",
),
]
pytestmark = pytest.mark.integration
def test_executor_model_completes_via_litellm() -> None:
missing = [var for var in ALSO_REQUIRED if not os.environ.get(var)]
assert not missing, (
f"{SENTINEL} is set, so the live LLM path is expected to work, "
f"but these are missing: {', '.join(missing)}"
)
client = LiteLLMClient.from_env()
def test_executor_model_completes_via_litellm(
litellm: LiteLLMClient, live_config: AgentConfig
) -> None:
events = [
StoredEvent(
id=1,
@@ -45,9 +23,39 @@ def test_executor_model_completes_via_litellm() -> None:
),
)
]
result = client.complete(
events, tools=None, model=os.environ["NISSE_ROLE_EXECUTOR"]
)
result = litellm.complete(events, tools=None, model=live_config.roles.executor)
texts = [e.text for e in result if isinstance(e, AssistantText)]
assert texts, f"no assistant text in {result!r}"
assert "pong" in texts[0].lower()
def test_declared_limits_reach_a_budgeted_request(
litellm: LiteLLMClient, live_config: AgentConfig
) -> None:
"""The whole chain: LiteLLM's model_info, the catalog, the budget, and a
real request through the proxy with the ceiling the budget chose."""
catalog = ModelCatalog.discover(live_config.llm_base_url, live_config.llm_token)
roles = live_config.roles.pinned()
catalog.require_limits(roles) # every test role has a declared window
model = live_config.roles.executor
facts = catalog.facts(model)
client = BudgetedClient(
litellm,
lambda m: (
catalog.facts(m).max_input_tokens,
catalog.facts(m).max_output_tokens,
),
max_output_tokens=live_config.max_output_tokens,
)
assert client.budget_for(model).window == facts.max_input_tokens
events = [
StoredEvent(
id=1, ts="t", event=UserMessage(text="Reply with exactly: pong", source="t")
)
]
produced = client.complete(events, None, model, system="Be terse.")
texts = [e.text for e in produced if isinstance(e, AssistantText)]
assert texts and "pong" in texts[0].lower()
assert client.last_usage.get("input", 0) > 0
tests/integration/test_matrix_live.py
+6 -33
View File
@@ -1,36 +1,14 @@
"""Live smoke test against the real homeserver, bound strictly to the
dedicated test room (never the production room).
dedicated test room that nisse-test.yaml names (never the production room)."""
NISSE_MATRIX_HOMESERVER alone decides whether this runs: if it is set, the
live path is expected to work, and anything else missing is a *failure*, not
a skip. A half-configured environment must not pass quietly as a skip — that
is how a broken CI credential hides."""
import os
import uuid
from pathlib import Path
import pytest
from nisse_dev.envfile import load_env_file
from nisse_dev.config import AgentConfig
from nisse_dev.matrix import MatrixConfig, MatrixTransport
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_MATRIX_HOMESERVER"
ALSO_REQUIRED = (
"NISSE_MATRIX_USER_ID",
"NISSE_MATRIX_TOKEN",
"NISSE_MATRIX_TEST_ROOM_ID",
)
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no live homeserver configured ({SENTINEL} unset)",
),
]
pytestmark = pytest.mark.integration
def _assert_testing_identity(transport: MatrixTransport, config: MatrixConfig) -> None:
@@ -46,7 +24,7 @@ def _assert_testing_identity(transport: MatrixTransport, config: MatrixConfig) -
whoami.raise_for_status()
actual = whoami.json()["user_id"]
assert actual == config.user_id, (
f"NISSE_MATRIX_USER_ID is {config.user_id} but the token belongs to "
f"matrix.user_id is {config.user_id} but the token belongs to "
f"{actual}; the self-message filter would not match"
)
assert "testing" in actual, (
@@ -54,13 +32,8 @@ def _assert_testing_identity(transport: MatrixTransport, config: MatrixConfig) -
)
def test_send_lands_in_test_room() -> None:
missing = [var for var in ALSO_REQUIRED if not os.environ.get(var)]
assert not missing, (
f"{SENTINEL} is set, so the live Matrix path is expected to work, "
f"but these are missing: {', '.join(missing)}"
)
config = MatrixConfig.from_env(room_var="NISSE_MATRIX_TEST_ROOM_ID")
def test_send_lands_in_test_room(live_config: AgentConfig) -> None:
config = live_config.matrix
marker = f"transport smoke test {uuid.uuid4().hex}"
transport = MatrixTransport(config)
try:
tests/integration/test_milestone_live.py
+8 -22
View File
@@ -2,12 +2,11 @@
approve through the gate, then let the executor role actually create and test
a file on disk.
Gated on NISSE_LIVE_PLANNER_MODEL like the planning test — it needs a model
that holds the tool-call contract on both ends.
Uses the test config's planner model for every role: it needs a model that
holds the tool-call contract on both ends.
"""
import io
import os
from pathlib import Path
import pytest
@@ -19,7 +18,6 @@ from nisse_dev.control import (
EXECUTING,
Orchestrator,
)
from nisse_dev.envfile import load_env_file
from nisse_dev.executor import LLMExecutor
from nisse_dev.llm import LiteLLMClient, RoleMap
from nisse_dev.oplog import OpLog
@@ -28,17 +26,7 @@ from nisse_dev.state import StateStore
from nisse_dev.tools import Workspace
from nisse_dev.transport import InMemoryTransport
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_LIVE_PLANNER_MODEL"
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no live planner route configured ({SENTINEL} unset)",
),
]
pytestmark = pytest.mark.integration
def _drive_to_execution(
@@ -67,16 +55,14 @@ def _drive_to_execution(
)
def test_plan_approve_execute_produces_a_real_file(tmp_path: Path) -> None:
assert os.environ.get("NISSE_LLM_TOKEN"), (
f"{SENTINEL} is set, so the live path is expected to work, "
"but NISSE_LLM_TOKEN is missing"
)
def test_plan_approve_execute_produces_a_real_file(
tmp_path: Path, litellm: LiteLLMClient, live_model: str
) -> None:
# Both roles use the capable model here: the point is the whole gated
# loop over a real proxy, not local-model competence (measured separately).
model = os.environ[SENTINEL]
model = live_model
roles = RoleMap(executor=model, planner=model, delegate=model, consultant=model)
client = LiteLLMClient.from_env()
client = litellm
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
workspace = Workspace(tmp_path / "work", timeout_s=60)
transport = InMemoryTransport()
Loaded 100 of 104 files, more files were not shown because too many files have changed in this diff. Show more