dongarandClaude Opus 5.5
154a0c3fa8
What isn't the dev agent's moves to nisse-core, unchanged in behaviour, with its tests: canonical, render, render_anthropic, llm, catalog, probe, context, views, state, reconcile, worker, cancel, dump, transport, matrix, execution, testing, vault, memory, oplog, envfile and gitea (whole). - nisse_core.state.StateStore holds the tables every agent has; an agent's subclass adds its tables (version, migrations), roles and views. nisse_dev.state.StateStore adds the build VM blocks (v5). - views.project takes what each role is shown, rather than a fixed table. - Vault takes the agent's name (agent=), for provenance. - A test that nisse_core never imports an agent. - The loop, config, git, delegation, expert, tools and evals stay in nisse-dev until nisse-ops has a loop to share. Docs: the Development Plan marks M7.1 done and notes that Phase 6 waits on the agent host and Terraform; Architecture Part L names the package. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
35 lines
1.1 KiB
Python
35 lines
1.1 KiB
Python
"""One live benchmark run: the simplest task, through the agent's own
|
|
client wiring, with the test config's driver."""
|
|
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from nisse_core.catalog import ModelCatalog
|
|
from nisse_core.execution import SSHHost
|
|
from nisse_dev.bench.runner import run_task
|
|
from nisse_dev.bench.tasks import load_tasks
|
|
from nisse_dev.config import AgentConfig
|
|
from nisse_dev.loop.build import make_llm_clients
|
|
|
|
pytestmark = pytest.mark.integration
|
|
|
|
|
|
def test_the_simplest_task_passes_live(
|
|
live_config: AgentConfig, tmp_path: Path, ssh_host: SSHHost
|
|
) -> None:
|
|
catalog = ModelCatalog.discover(live_config.llm_base_url, live_config.llm_token)
|
|
normalized, _ = make_llm_clients(live_config, catalog)
|
|
facts = catalog.facts(live_config.roles.driver)
|
|
[task] = load_tasks(["write-function"])
|
|
|
|
record = run_task(
|
|
task, facts, normalized, live_config.tuning, host=ssh_host, workdir=tmp_path
|
|
)
|
|
|
|
assert record.fallbacks == []
|
|
assert record.success, (
|
|
f"{record.outcome}: {record.outcome_note}\n{record.check_output}"
|
|
)
|
|
assert record.cost >= 0
|