Development Plan M1.1, as discussed.
- Schema v4: a work_items table (id, scope, title, status, outcome,
position, data) and an agent row whose active_item points at one
(NULL: idle). Canonical events and intents carry their work item, and
events their actor. Older stores are refused with a clear message:
there was no production data, so no migration (the mechanism stays).
- The log is reached through a Transcript bound to one work item and
one actor: store.transcript(item, actor) reads only that item and
writes with both. The planner, executor, delegation and expert desk
get one per call from the orchestrator instead of holding the store,
so a component can neither read another task's events (the planner
read every earlier task's, #78) nor write without attribution.
store.all_events() is kept for readers that span items (evaluation
capture, dumps). Intents are begun through the transcript or with
work_item=, so each belongs to its item.
- The orchestrator opens a work item on 'use', closes it with its
outcome (completed or abandoned), and carries what the operator said
while idle into the new item as copies with their origin, so a task
sent before 'use' is still planned from (D1). control.py keeps its
state machine through get_state()/transition(), now over the active
work item, until M2.6 retires it.
- nisse_dev.dump shows work items and each event's item and actor.
Tests: test_work_items.py (scoping, attribution, intent ownership,
restart, outcomes, carried messages, a second task never seeing the
first); the suites updated for the per-call transcript.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Architecture L9's role names, for the two roles with a direct
counterpart: the delegation role is now 'helper' and the ask_expert role
'expert', in RoleMap, configuration (roles.helper, roles.expert),
recorded events and tests. 'planner' and 'executor' stay distinct for
now; they become 'driver' when the new loop merges them (Development
Plan M2.3).
Configurations must rename their keys: nisse-dev.example.yaml and
tests/integration/nisse-test.yaml are updated here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Calibration showed Haiku fixing the right constant but keeping only four
decimal places, which passes the visible test and still loses the JPY
rate's precision. An explicit plan should say so.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
All seven newer local models passed the original four tasks 20 of 20,
so the benchmark no longer told them apart. The new tasks keep plans
explicit, as the planner is told to write them, and make the work
itself harder:
- trace-bug: a wrong multi-currency total whose cause is a shared
rate helper; discounts and tax go through it too, so a fix at the
call site fails the check.
- refactor-rename: rename load_config across a ten-module package,
leaving load_config_defaults and reload_config alone.
- spec-parser: implement parse_duration from an eight-rule SPEC.md
given three examples, and write tests for every rule.
- long-milestone: eight dependent steps adding due dates to a task
tracker, from the model through storage versions to the CLI.
Self-tests prove that a call-site fix, a blind text replace and a
partial rename all fail their checks.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The report grouped runs by model, serving note and task version, so
runs made at different tuning were silently averaged together. Tuning
is now part of the key, and a tuning column shows how a row's settings
differ from today's defaults; it appears only when some row needs it.
tuning.executor.max_turns goes from 24 to 40. On the harder benchmark
tasks Haiku, the calibration ceiling, hit 24 on both refactor-rename
runs and both long-milestone runs, one of which had finished the work.
The cap guards against a model trying forever; it isn't a budget.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.
Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.
- Four canned milestones under nisse_dev/bench/tasks: write a function,
fix a bug a failing test exposes, thread an option through several
files, and find one failure in output far past max_output_chars. Each
is an approved plan over a seed project, judged by a hidden check the
model never sees; a reference solution proves each is solvable and
that its check tells the difference
- Runs use the agent's own client wiring, now shared as
make_llm_clients, with delegation and ask_expert off: an escalation is
a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
the model's backing name and declared facts, a note on how it was
served, the task's content hash, the agent's commit and tuning, the
outcome, turns, misbehaviour counts, context trims, tokens, time and
LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
a new model through the context probe, the capability matrix and the
benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
requests it had to trim
- The tuning guide gains "Benchmarking models"
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dongarandClaude Opus 5.5