Benchmark: workspaces are git repositories #177

2026-10-24T04:44:10Z

dev-agent commented

2026-10-24 04:44:10 +00:00

Benchmark workspaces now start as a git repository, with the task's seed files as the first commit on main, the same as the clone the agent works in.

Why: the workspaces were plain directories, so a model that opens with git log or git status got exit code 128. Two failed commands in a row stop a run as "stuck". In the 2026-10-24 runs this caused 10 of Flash-Next (thinking)'s 18 failures, 3 of lowthink's 9, 8 of nothink's 28 and none of qwen3.8-27b's 4. (The commit message says "10 of 13", counted partway through the run.)

What changes:

bench/tasks.py: GIT_SEED initialises and commits the seed. It runs after local seeding (seed_into) and on the build VM (seed_workspace, which raises if it fails).
How a workspace is set up (SEED_SETUP) is hashed into each task's content_hash, so earlier runs show as not comparable rather than mixing with new ones.

Tests (nisse-dev/tests/test_bench.py):

the seed is a git repository with one clean commit;
a change to how the seed is set up changes the hash;
on the build VM, the model can read the history (git log, git status) without a failed command. This one fails without the fix.

After merge I'll rerun Flash-Next thinking and lowthink on the new workspaces and put the comparison in a separate PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

Benchmark workspaces now start as a git repository, with the task's seed files as the first commit on `main`, the same as the clone the agent works in. **Why:** the workspaces were plain directories, so a model that opens with `git log` or `git status` got exit code 128. Two failed commands in a row stop a run as "stuck". In the 2026-10-24 runs this caused 10 of Flash-Next (thinking)'s 18 failures, 3 of lowthink's 9, 8 of nothink's 28 and none of qwen3.8-27b's 4. (The commit message says "10 of 13", counted partway through the run.) **What changes:** - `bench/tasks.py`: `GIT_SEED` initialises and commits the seed. It runs after local seeding (`seed_into`) and on the build VM (`seed_workspace`, which raises if it fails). - How a workspace is set up (`SEED_SETUP`) is hashed into each task's `content_hash`, so earlier runs show as not comparable rather than mixing with new ones. **Tests** (`nisse-dev/tests/test_bench.py`): - the seed is a git repository with one clean commit; - a change to how the seed is set up changes the hash; - on the build VM, the model can read the history (`git log`, `git status`) without a failed command. This one fails without the fix. After merge I'll rerun Flash-Next thinking and lowthink on the new workspaces and put the comparison in a separate PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

dev-agent added 1 commit 2026-10-24 04:44:10 +00:00

Benchmark: the seed is a git repository's first commit

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 3m33s

Details

CI / e2e (pull_request) Successful in 4m28s

Details

7a57cef9ff

The agent works in a clone, but benchmark workspaces were plain
directories, so a model that starts with `git log` failed two commands
in a row and was stopped as stuck within seconds (Flash-Next, 10 of 13
failures on 2026-10-24). The seed is now committed on main, locally and
on the build VM, and how a workspace is set up is part of each task's
hash, so earlier runs are reported as not comparable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

dev-agent referenced this pull request

2026-10-24 08:39:50 +00:00

The call record: every model call, for the operator's study (M6.6a) #179

dev-agent referenced this issue from a commit

2026-10-24 08:40:24 +00:00

Benchmark Qwen3.8-Flash-Next against qwen3.8-27b

dev-agent referenced this pull request

2026-10-24 08:40:36 +00:00

Benchmark Qwen3.8-Flash-Next against qwen3.8-27b #180

dongar approved these changes

2026-10-24 13:08:02 +00:00

dev-agent merged commit 6f07507420 into main