Keep runs with different tuning apart in the report, and raise max_turns to 40 #52

2026-09-27T22:19:58Z

dev-agent commented

2026-09-27 22:19:58 +00:00

Report: tuning is part of each row's identity

Every run already records its tuning, but summarise grouped runs only by model, serving note and task version. Runs at max_turns: 24 and 40 would have been averaged into one row without any sign of it.

bench/results.py: the grouping key includes tuning_label(tuning), which lists how a run's settings differ from today's defaults, e.g. executor.max_turns=24. Runs at today's settings have an empty label.
The report shows a tuning column, with a footnote, only when some row differs, so a report at current settings looks as before.

`tuning.executor.max_turns`: 24 → 40

On the harder benchmark tasks, Haiku (the calibration ceiling) hit the 24-turn cap on both runs of refactor-rename, which it does by editing ten files one at a time, and both runs of long-milestone. One of the long-milestone runs had actually finished the work. You asked for the cap to be only a guard against a model trying forever, so it's raised to 40:

in the default in config.py
in nisse-dev.example.yaml
in the Tuning Guide, with what was measured

The final value may move once Haiku's calibration at 40 shows how many turns successful runs take. The guide's rule of thumb is 1.5–2× the longer successful runs.

Tests

Two new tests: runs with different tuning get separate rows labelled with the difference, and a report at default tuning has no tuning column.
Three tests asserting the old default of 24 are updated to 40.

CI steps run locally: ruff check, ruff format --check, and pytest -m "not integration": 405 passed. One run out of eight had a single failure I couldn't reproduce or identify; seven runs in a row were clean. If CI shows it, I'll chase it.

🤖 Generated with Claude Code

## Report: tuning is part of each row's identity Every run already records its tuning, but `summarise` grouped runs only by model, serving note and task version. Runs at `max_turns: 24` and `40` would have been averaged into one row without any sign of it. - **`bench/results.py`:** the grouping key includes `tuning_label(tuning)`, which lists how a run's settings differ from today's defaults, e.g. `executor.max_turns=24`. Runs at today's settings have an empty label. - **The report** shows a **tuning** column, with a footnote, only when some row differs, so a report at current settings looks as before. ## `tuning.executor.max_turns`: 24 → 40 On the harder benchmark tasks, Haiku (the calibration ceiling) hit the 24-turn cap on both runs of `refactor-rename`, which it does by editing ten files one at a time, and both runs of `long-milestone`. One of the `long-milestone` runs had actually finished the work. You asked for the cap to be only a guard against a model trying forever, so it's raised to 40: - in the default in `config.py` - in `nisse-dev.example.yaml` - in the Tuning Guide, with what was measured The final value may move once Haiku's calibration at 40 shows how many turns successful runs take. The guide's rule of thumb is 1.5–2× the longer successful runs. ## Tests - **Two new tests:** runs with different tuning get separate rows labelled with the difference, and a report at default tuning has no tuning column. - **Three tests asserting the old default of 24** are updated to 40. CI steps run locally: `ruff check`, `ruff format --check`, and `pytest -m "not integration"`: 405 passed. One run out of eight had a single failure I couldn't reproduce or identify; seven runs in a row were clean. If CI shows it, I'll chase it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dongar approved these changes

2026-09-27 22:24:31 +00:00

Dismissed

dev-agent added 1 commit 2026-09-27 22:25:40 +00:00

Keep runs with different tuning apart in the report, and raise max_turns to 40

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m11s

Details

f881a18ca4

The report grouped runs by model, serving note and task version, so
runs made at different tuning were silently averaged together. Tuning
is now part of the key, and a tuning column shows how a row's settings
differ from today's defaults; it appears only when some row needs it.

tuning.executor.max_turns goes from 24 to 40. On the harder benchmark
tasks Haiku, the calibration ceiling, hit 24 on both refactor-rename
runs and both long-milestone runs, one of which had finished the work.
The cap guards against a model trying forever; it isn't a budget.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dev-agent force-pushed report-keeps-tuning-apart from 570da5784e to f881a18ca4

2026-09-27 22:25:40 +00:00

Compare

dev-agent dismissed dongar's review 2026-09-27 22:25:40 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

dongar approved these changes

2026-09-27 22:43:23 +00:00

dongar merged commit f2c48d8b16 into main

2026-09-27 22:43:26 +00:00

dongar deleted branch report-keeps-tuning-apart

2026-09-27 22:43:26 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#52

Keep runs with different tuning apart in the report, and raise max_turns to 40 #52

Report: tuning is part of each row's identity

tuning.executor.max_turns: 24 → 40

Tests

`tuning.executor.max_turns`: 24 → 40