Run every command on the build VM over SSH (M1.3)
CI / checks (push) Skipped
CI / checks (pull_request) Failing after 55s

execution.SSHHost.execute is now the only way the agent runs anything
(L13). Commands travel as a script on SSH's stdin, so the secrets handed
to a call are never on a command line, and every value handed out is
redacted from the output. Timeouts kill the remote process group, and
an unreachable host is reported as such, not as a failed command.

Workspace and GitRepo run on the build VM. The clone uses the
repository's SSH URL, so no credential is stored in it (#89); fetches
and pushes get the push key over stdin, in a short-lived ssh-agent,
and PRs are still opened from the agent's host with its API token.
The bench runs on the build host too.

Tests reach "the build VM" through a throwaway local sshd (nisse_dev
.testing); CI installs openssh-server. The live push test now pushes
with the sandbox's deploy key (NISSE_PUSH_KEY).

Fixes #89.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-10-02 21:39:46 +00:00
1 parent b7f1f3371c
commit 761ebdb857
32 files changed
+1098 -280

No files matched your search

+10 -3
View File
@@ -1,9 +1,11 @@
"""Running one task once with one executor model, and recording it."""
import io
import shlex
import subprocess
import tempfile
import time
import uuid
from dataclasses import asdict, dataclass, field
from datetime import UTC, datetime
from pathlib import Path
@@ -20,6 +22,7 @@ from nisse_dev.canonical import (
)
from nisse_dev.catalog import ModelFacts
from nisse_dev.config import Tuning
from nisse_dev.execution import SSHHost
from nisse_dev.executor import LLMExecutor
from nisse_dev.llm import RoleMap
from nisse_dev.oplog import OpLog
@@ -150,6 +153,7 @@ def run_task(
client: Any,
tuning: Tuning,
*,
host: SSHHost,
note: str = "",
commit: str = "",
workdir: Path | None = None,
@@ -161,12 +165,14 @@ def run_task(
model under test, not a frontier model's success."""
with tempfile.TemporaryDirectory(dir=workdir) as tmp:
root = Path(tmp)
# The workspace is on the build VM, like the agent's own (L13).
workspace = Workspace(
root / "work",
host,
f"/tmp/nisse-bench/{uuid.uuid4().hex}",
timeout_s=tuning.tools.command_timeout_s,
max_output_chars=tuning.tools.max_output_chars,
)
task.seed_into(workspace.root)
task.seed_workspace(workspace)
store = StateStore(root / "state.db", oplog=OpLog("nisse-bench", io.StringIO()))
# One work item per run, as the agent's own tasks are (L1).
store.open_work_item(
@@ -198,7 +204,8 @@ def run_task(
status, detail = "error", repr(exc)
wall = time.monotonic() - started
success, check_output = task.run_check(workspace.root)
success, check_output = task.check_workspace(workspace)
host.execute(f"rm -rf -- {shlex.quote(str(workspace.root))}", timeout_s=60)
failed, invented = _count(store.transcript("bench", "executor").read())
store.close()