The call record: every model call, for the operator's study (M6.6a)
nisse-core records each model call of every agent into a file beside its state store (<state>.calls.db), for the status page (L7, X6): - the request body exactly as sent (never headers), the raw response with its reasoning, the model asked and the one that answered, time, tokens and llama-server's own prompt reuse (timings.cache_n); - its work item, role, run and purpose: Driver.run and nisse-dev's delegation loop open runs, nested; compaction, expert consults, the classifier and the go-ahead reader label their calls; - provenance, worked out in BudgetedClient by comparing the events at each stage: each event sent with its log id and content as sent, which a compaction replaced, which fitting shortened (by how much) and dropped. Content is stored once by hash, compressed. It is diagnostic only: the agent never reads it, a failed write is logged once and recording stops while the call goes on, and the file is deleted when the agent starts, so its layout never migrates; CallReader refuses a layout it doesn't know. While recording, the system prompt ends with one sentence saying so. `call_record: false` in either agent's YAML turns it off. The benchmark records each run beside that run's own state store, and `--keep-runs DIR` keeps both for the page. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
This commit is contained in:
1 parent
cba35eb7d4
commit
dd060566b2
22 files changed
+1346
-48
No files matched your search
@@ -573,3 +573,28 @@ def test_the_vault_is_read() -> None:
|
||||
def test_a_bad_vault_is_refused(vault: dict[str, Any], said: str) -> None:
|
||||
with pytest.raises(ConfigError, match=re.escape(said)):
|
||||
build(vault=vault)
|
||||
|
||||
|
||||
def test_the_call_record_is_on_unless_turned_off() -> None:
|
||||
assert build().call_record is True
|
||||
assert build(call_record=False).call_record is False
|
||||
with pytest.raises(ConfigError, match="call_record"):
|
||||
build(call_record="no")
|
||||
|
||||
|
||||
def test_both_routes_record_into_one_file_beside_the_state_store(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
from nisse_core.models.catalog import ModelCatalog
|
||||
from nisse_dev.loop.build import make_llm_clients
|
||||
|
||||
normalized, passthrough = make_llm_clients(
|
||||
build(state_db=str(tmp_path / "state.db")), ModelCatalog({})
|
||||
)
|
||||
assert normalized.call_recorder is passthrough.call_recorder
|
||||
assert normalized.call_recorder.path == tmp_path / "state.calls.db"
|
||||
off, _ = make_llm_clients(
|
||||
build(state_db=str(tmp_path / "other.db"), call_record=False), ModelCatalog({})
|
||||
)
|
||||
assert off.call_recorder is None
|
||||
assert not (tmp_path / "other.calls.db").exists()
|
||||
Reference in new issue
Block a user