Runs that go in circles end (M7.8) #183
No files matched your search
@@ -811,7 +811,12 @@ every agent. Seen 2026-10-24 on Flash-Next: invented tool names met with the
|
|||||||
prose reminder turn after turn, and a no-thinking run posting the same apology
|
prose reminder turn after turn, and a no-thinking run posting the same apology
|
||||||
comment about 20 times until the turn cap. Repeated near-identical turns, or the
|
comment about 20 times until the turn cap. Repeated near-identical turns, or the
|
||||||
same correction given too often, end the run as escalated, with what it was
|
same correction given too often, end the run as escalated, with what it was
|
||||||
doing; the limits are tuning.
|
doing; the limits are tuning. **Done** (this PR): `RunOutcome("circling")`
|
||||||
|
after the same calls getting much the same back, one correction, or apologies,
|
||||||
|
`tuning.*.circle_repeats` turns in a row (3). Near-identical calls only flag on
|
||||||
|
the page (`git add a.py`, `b.py`). nisse-dev calls in a helper as for a stuck
|
||||||
|
run, then asks; nisse-review counts a failed run and posts none of its drafts.
|
||||||
|
The helper's own loop keeps its caps.
|
||||||
|
|
||||||
### Phase 8 — `nisse-ops`
|
### Phase 8 — `nisse-ops`
|
||||||
|
|
||||||
|
|||||||
@@ -193,6 +193,28 @@ looking around, not doing the work. Before M2.8 this was
|
|||||||
(escalation is doing its job). Second, how often the local model recovers
|
(escalation is doing its job). Second, how often the local model recovers
|
||||||
on its own after its second failure (the limit may be too low).
|
on its own after its second failure (the limit may be too low).
|
||||||
|
|
||||||
|
### `tuning.driver.circle_repeats` — default 3, and `circle_similar` — default 0.9
|
||||||
|
|
||||||
|
When a driver run going in circles ends (architecture L2, M7.8), escalated as
|
||||||
|
for `failure_limit`: a helper first, then you. A run circles when it makes
|
||||||
|
the same tool calls this many turns in a row and gets much the same back
|
||||||
|
each time, when the harness gives one correction this many times ("no such
|
||||||
|
tool: …"), or when it apologises this many turns in a row. `circle_similar`
|
||||||
|
is how alike two replies' text and two turns' results must be (0 to 1). 0
|
||||||
|
for `circle_repeats` never ends a run this way; 1 isn't allowed, since every
|
||||||
|
run would end at its first turn.
|
||||||
|
|
||||||
|
- **Why only the same calls:** `git add a.py`, then `b.py`, then `c.py` are
|
||||||
|
near-identical and each gets the same empty answer, but they are progress.
|
||||||
|
The status page flags near-identical turns too, and scattered apologies;
|
||||||
|
the driver ends a run only on what can't be progress.
|
||||||
|
- **Too low:** a model checking the same thing twice is cut off. 2 is the
|
||||||
|
least that means anything.
|
||||||
|
- **Too high:** a looping run spends that many more turns, and context.
|
||||||
|
- **Measured:** Qwen3.8-Flash-Next (2026-10-24) posted the same apology
|
||||||
|
comment about 20 times until the turn cap; 3 would have ended it at the
|
||||||
|
third.
|
||||||
|
|
||||||
### `tuning.preflight.max_turns` — default 20
|
### `tuning.preflight.max_turns` — default 20
|
||||||
|
|
||||||
How many model calls one toolchain step may make (D3). The harness starts one
|
How many model calls one toolchain step may make (D3). The harness starts one
|
||||||
@@ -673,6 +695,12 @@ Failed review runs on one PR, at one head, before the reviewer says on the PR
|
|||||||
that it couldn't review it, and stops trying until the next push or an
|
that it couldn't review it, and stops trying until the next push or an
|
||||||
operator's `review` comment.
|
operator's `review` comment.
|
||||||
|
|
||||||
|
### `tuning.review.circle_repeats` — default 3, and `circle_similar` — default 0.9
|
||||||
|
|
||||||
|
As `tuning.driver.circle_repeats`, for review runs. A review run going in
|
||||||
|
circles is a failed run (it counts toward `attempts`), and what it drafted is
|
||||||
|
dropped rather than posted: the drafts are the repeats.
|
||||||
|
|
||||||
### `tuning.review.ci_wait_s` — default 3600
|
### `tuning.review.ci_wait_s` — default 3600
|
||||||
|
|
||||||
How long a new head's review waits while its checks are pending. A review
|
How long a new head's review waits while its checks are pending. A review
|
||||||
|
|||||||
@@ -146,6 +146,16 @@ def parse_tuning(
|
|||||||
return tuning(**sections)
|
return tuning(**sections)
|
||||||
|
|
||||||
|
|
||||||
|
def check_circles(repeats: int, similar: float, where: str, errors: list[str]) -> None:
|
||||||
|
"""A run ends going in circles (L2, M7.8) at ``repeats`` alike turns or
|
||||||
|
corrections: 0 turns that off, and 1 would end every run at its first
|
||||||
|
turn. ``similar`` is how alike two replies are, at most 1."""
|
||||||
|
if repeats == 1:
|
||||||
|
errors.append(f"{where}.circle_repeats: must be 0 (off) or 2 or more")
|
||||||
|
if similar > 1:
|
||||||
|
errors.append(f"{where}.circle_similar: must be at most 1")
|
||||||
|
|
||||||
|
|
||||||
def llm_numbers(values: Mapping[str, Any], errors: list[str]) -> tuple[int, float]:
|
def llm_numbers(values: Mapping[str, Any], errors: list[str]) -> tuple[int, float]:
|
||||||
"""`llm.max_output_tokens` and `llm.request_timeout_s`, or their
|
"""`llm.max_output_tokens` and `llm.request_timeout_s`, or their
|
||||||
defaults."""
|
defaults."""
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
"""Runs going in circles (L2, X6): turns that repeat near-identically, one
|
"""Runs going in circles (L2, X6): turns that repeat near-identically, one
|
||||||
correction given again and again, or apologies over and over.
|
correction given again and again, or apologies over and over.
|
||||||
|
|
||||||
A pure function over a run's turns, so the status page can flag a loop now
|
A pure function over a run's turns, so the status page flags a loop and
|
||||||
and ``Driver.run`` can end one later (M7.8) from the same rules.
|
``Driver.run`` ends one (M7.8) from the same rules.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import re
|
import re
|
||||||
@@ -29,6 +29,10 @@ class Turn:
|
|||||||
text: str = ""
|
text: str = ""
|
||||||
calls: tuple[tuple[str, str], ...] = () # (tool, arguments as sent)
|
calls: tuple[tuple[str, str], ...] = () # (tool, arguments as sent)
|
||||||
told: tuple[str, ...] = () # corrections given since the turn before
|
told: tuple[str, ...] = () # corrections given since the turn before
|
||||||
|
# What its calls got back. Two turns alike in their calls but not in
|
||||||
|
# what they got back, exactly (reading sibling files), are progress, not
|
||||||
|
# a circle; the driver gives them, the page doesn't yet.
|
||||||
|
results: tuple[str, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -39,10 +43,17 @@ class Circle:
|
|||||||
|
|
||||||
|
|
||||||
def find_circles(
|
def find_circles(
|
||||||
turns: Sequence[Turn], *, similar: float = SIMILAR, repeats: int = REPEATS
|
turns: Sequence[Turn],
|
||||||
|
*,
|
||||||
|
similar: float = SIMILAR,
|
||||||
|
repeats: int = REPEATS,
|
||||||
|
exact_calls: bool = False,
|
||||||
) -> list[Circle]:
|
) -> list[Circle]:
|
||||||
"""Every circle in a run, in the order they started."""
|
"""Every circle in a run, in the order they started. ``exact_calls``:
|
||||||
found = _repeated_replies(turns, similar, repeats)
|
replies repeat only when their tool calls are exactly the same, as the
|
||||||
|
driver judges before it ends a run; near-identical is enough for the
|
||||||
|
page to flag one (``git add a.py`` then ``b.py`` are near-identical)."""
|
||||||
|
found = _repeated_replies(turns, similar, repeats, exact_calls)
|
||||||
found += _repeated_corrections(turns, repeats)
|
found += _repeated_corrections(turns, repeats)
|
||||||
apologies = tuple(i for i, t in enumerate(turns) if _APOLOGY.search(t.text))
|
apologies = tuple(i for i, t in enumerate(turns) if _APOLOGY.search(t.text))
|
||||||
if len(apologies) >= repeats:
|
if len(apologies) >= repeats:
|
||||||
@@ -61,6 +72,33 @@ def _signature(turn: Turn) -> str:
|
|||||||
return f"{turn.text.strip()}\n{calls}"[:_COMPARED]
|
return f"{turn.text.strip()}\n{calls}"[:_COMPARED]
|
||||||
|
|
||||||
|
|
||||||
|
def _same_turn(a: Turn, b: Turn, similar: float, exact_calls: bool) -> bool:
|
||||||
|
"""The same reply getting the same back. A model repeating itself gets
|
||||||
|
much the same answer each time ("noted (2 in the draft)"); alike calls
|
||||||
|
that got different answers are progress (two files of near-identical
|
||||||
|
contents are still two files)."""
|
||||||
|
if exact_calls:
|
||||||
|
joined = ("\n".join(t.results)[:_COMPARED] for t in (a, b))
|
||||||
|
return (
|
||||||
|
a.calls == b.calls
|
||||||
|
and _alike(*joined, similar)
|
||||||
|
and _alike(a.text.strip(), b.text.strip(), similar)
|
||||||
|
)
|
||||||
|
return a.results == b.results and _alike(_signature(a), _signature(b), similar)
|
||||||
|
|
||||||
|
|
||||||
|
def apologies_in_a_row(turns: Sequence[Turn], repeats: int = REPEATS) -> Circle | None:
|
||||||
|
"""The last ``repeats`` turns all apologising: a model saying sorry
|
||||||
|
again and again instead of moving on, as the driver ends a run for.
|
||||||
|
Apologies spread over a run are only flagged (``find_circles``)."""
|
||||||
|
last = range(len(turns) - repeats, len(turns))
|
||||||
|
if repeats <= 0 or last.start < 0:
|
||||||
|
return None
|
||||||
|
if all(_APOLOGY.search(_signature(turns[i])) for i in last):
|
||||||
|
return Circle("repeated apology", tuple(last), f"{repeats} apologies in a row")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _alike(a: str, b: str, similar: float) -> bool:
|
def _alike(a: str, b: str, similar: float) -> bool:
|
||||||
if a == b:
|
if a == b:
|
||||||
return True
|
return True
|
||||||
@@ -69,7 +107,7 @@ def _alike(a: str, b: str, similar: float) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
def _repeated_replies(
|
def _repeated_replies(
|
||||||
turns: Sequence[Turn], similar: float, repeats: int
|
turns: Sequence[Turn], similar: float, repeats: int, exact_calls: bool
|
||||||
) -> list[Circle]:
|
) -> list[Circle]:
|
||||||
"""Stretches of consecutive turns each alike the one before."""
|
"""Stretches of consecutive turns each alike the one before."""
|
||||||
found: list[Circle] = []
|
found: list[Circle] = []
|
||||||
@@ -79,7 +117,7 @@ def _repeated_replies(
|
|||||||
if (
|
if (
|
||||||
i < len(turns)
|
i < len(turns)
|
||||||
and signatures[i].strip()
|
and signatures[i].strip()
|
||||||
and _alike(signatures[i - 1], signatures[i], similar)
|
and _same_turn(turns[i - 1], turns[i], similar, exact_calls)
|
||||||
):
|
):
|
||||||
continue
|
continue
|
||||||
if i - start >= repeats:
|
if i - start >= repeats:
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
"""A driver run (L2): one model with tools, looping until it calls a waiting
|
"""A driver run (L2): one model with tools, looping until it calls a waiting
|
||||||
tool, runs out of turns, or is stopped.
|
tool, runs out of turns, goes in circles, or is stopped.
|
||||||
|
|
||||||
Everything the run does is written to the item's transcript as the driver
|
Everything the run does is written to the item's transcript as the driver
|
||||||
(L7), and every model call sees the driver's view of it (L6), compacted
|
(L7), and every model call sees the driver's view of it (L6), compacted
|
||||||
@@ -9,11 +9,13 @@ here, and a waiting one ends the run, so the scheduler can put it to the
|
|||||||
operator. Which tools wait is the agent's: each run is told.
|
operator. Which tools wait is the agent's: each run is told.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
from collections.abc import Callable, Collection, Mapping
|
from collections.abc import Callable, Collection, Mapping
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Any, Literal, Protocol
|
from typing import Any, Literal, Protocol
|
||||||
|
|
||||||
from nisse_core.log.canonical import AssistantText, Fact, ToolCall, ToolResult
|
from nisse_core.log.canonical import AssistantText, Fact, ToolCall, ToolResult
|
||||||
|
from nisse_core.loop import circles
|
||||||
from nisse_core.loop.tools import names
|
from nisse_core.loop.tools import names
|
||||||
from nisse_core.models import calls
|
from nisse_core.models import calls
|
||||||
from nisse_core.models.compact import Compactor
|
from nisse_core.models.compact import Compactor
|
||||||
@@ -47,8 +49,12 @@ class Executor(Protocol):
|
|||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class RunOutcome:
|
class RunOutcome:
|
||||||
# stuck: failed commands in a row hit the failure limit (L9's trigger);
|
# stuck: failed commands in a row hit the failure limit (L9's trigger);
|
||||||
# unreachable: the tool VM went out of reach (#137), noted in ``note``
|
# circling: the run repeated itself, or was told the same thing too often
|
||||||
kind: Literal["wait", "capped", "stuck", "stopped", "failed", "unreachable"]
|
# (L2, M7.8); unreachable: the tool VM went out of reach (#137). What
|
||||||
|
# happened is in ``note``.
|
||||||
|
kind: Literal[
|
||||||
|
"wait", "capped", "stuck", "circling", "stopped", "failed", "unreachable"
|
||||||
|
]
|
||||||
tool: str = "" # the waiting tool called
|
tool: str = "" # the waiting tool called
|
||||||
arguments: dict[str, Any] | None = None
|
arguments: dict[str, Any] | None = None
|
||||||
note: str = ""
|
note: str = ""
|
||||||
@@ -86,7 +92,11 @@ class Driver:
|
|||||||
cancel: Cancel | None = None,
|
cancel: Cancel | None = None,
|
||||||
failure_limit: int = 0, # 0: no limit
|
failure_limit: int = 0, # 0: no limit
|
||||||
compactor: Compactor | None = None,
|
compactor: Compactor | None = None,
|
||||||
|
circle_repeats: int = circles.REPEATS, # 0: runs never end as circling
|
||||||
|
circle_similar: float = circles.SIMILAR,
|
||||||
) -> None:
|
) -> None:
|
||||||
|
self._circle_repeats = circle_repeats
|
||||||
|
self._circle_similar = circle_similar
|
||||||
self._client = client
|
self._client = client
|
||||||
self._compactor = compactor
|
self._compactor = compactor
|
||||||
self._model = model
|
self._model = model
|
||||||
@@ -140,6 +150,9 @@ class Driver:
|
|||||||
offered = names(tools)
|
offered = names(tools)
|
||||||
nudged = False
|
nudged = False
|
||||||
cap = max_turns or self._max_turns
|
cap = max_turns or self._max_turns
|
||||||
|
# Each turn as circles judge it (L2): the reply, what its calls got
|
||||||
|
# back, and the corrections the harness gave in answer.
|
||||||
|
turns: list[circles.Turn] = []
|
||||||
for turn in range(cap):
|
for turn in range(cap):
|
||||||
# L2: stop takes effect at the next tool boundary.
|
# L2: stop takes effect at the next tool boundary.
|
||||||
if self._cancel.requested:
|
if self._cancel.requested:
|
||||||
@@ -175,7 +188,10 @@ class Driver:
|
|||||||
# the user's side (models refuse assistant prefill), and the
|
# the user's side (models refuse assistant prefill), and the
|
||||||
# nudge is then in the log as what it was.
|
# nudge is then in the log as what it was.
|
||||||
transcript.as_actor("orchestrator").append(Fact(text=_PROSE_NUDGE))
|
transcript.as_actor("orchestrator").append(Fact(text=_PROSE_NUDGE))
|
||||||
|
turns.append(circles.Turn(text=said, told=(_PROSE_NUDGE,)))
|
||||||
continue
|
continue
|
||||||
|
results: list[str] = []
|
||||||
|
told: list[str] = [] # results the harness gave, not the tools
|
||||||
for index, call in enumerate(calls):
|
for index, call in enumerate(calls):
|
||||||
if self._cancel.requested:
|
if self._cancel.requested:
|
||||||
skip_calls(transcript, calls[index:])
|
skip_calls(transcript, calls[index:])
|
||||||
@@ -189,6 +205,8 @@ class Driver:
|
|||||||
transcript.append(
|
transcript.append(
|
||||||
ToolResult(call_id=call.call_id, result=problem, success=False)
|
ToolResult(call_id=call.call_id, result=problem, success=False)
|
||||||
)
|
)
|
||||||
|
results.append(problem)
|
||||||
|
told.append(problem)
|
||||||
skip_calls(
|
skip_calls(
|
||||||
transcript, calls[index + 1 :], "not run: fix the call above"
|
transcript, calls[index + 1 :], "not run: fix the call above"
|
||||||
)
|
)
|
||||||
@@ -233,6 +251,7 @@ class Driver:
|
|||||||
+ ", ".join(sorted(offered)),
|
+ ", ".join(sorted(offered)),
|
||||||
False,
|
False,
|
||||||
)
|
)
|
||||||
|
told.append(result)
|
||||||
else:
|
else:
|
||||||
screening = screen(call) if screen else Screening("run")
|
screening = screen(call) if screen else Screening("run")
|
||||||
if screening.decision == "ask":
|
if screening.decision == "ask":
|
||||||
@@ -262,6 +281,8 @@ class Driver:
|
|||||||
success=False,
|
success=False,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
results.append(f"not run: {screening.text}")
|
||||||
|
told.append(f"not run: {screening.text}")
|
||||||
skip_calls(
|
skip_calls(
|
||||||
transcript,
|
transcript,
|
||||||
calls[index + 1 :],
|
calls[index + 1 :],
|
||||||
@@ -281,6 +302,7 @@ class Driver:
|
|||||||
transcript.append(
|
transcript.append(
|
||||||
ToolResult(call_id=call.call_id, result=result, success=ok)
|
ToolResult(call_id=call.call_id, result=result, success=ok)
|
||||||
)
|
)
|
||||||
|
results.append(result)
|
||||||
if call.tool == "run_command":
|
if call.tool == "run_command":
|
||||||
failures = [] if ok else [*failures, _brief(call, result)]
|
failures = [] if ok else [*failures, _brief(call, result)]
|
||||||
if self._failure_limit and len(failures) >= self._failure_limit:
|
if self._failure_limit and len(failures) >= self._failure_limit:
|
||||||
@@ -292,8 +314,53 @@ class Driver:
|
|||||||
note=f"{len(failures)} commands failed in a row: "
|
note=f"{len(failures)} commands failed in a row: "
|
||||||
+ "; ".join(failures),
|
+ "; ".join(failures),
|
||||||
)
|
)
|
||||||
|
turns.append(
|
||||||
|
circles.Turn(
|
||||||
|
text=said,
|
||||||
|
calls=tuple((c.tool, _args(c)) for c in calls),
|
||||||
|
told=tuple(told),
|
||||||
|
results=tuple(results),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
circling = self._circling(turns)
|
||||||
|
if circling is not None:
|
||||||
|
# In the log as the harness's, for whoever picks it up next.
|
||||||
|
transcript.as_actor("orchestrator").append(
|
||||||
|
Fact(
|
||||||
|
text=f"The run was ended: it was going in circles ({circling})."
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return RunOutcome("circling", note=circling)
|
||||||
return RunOutcome("capped", note=f"hit the {cap}-turn cap")
|
return RunOutcome("capped", note=f"hit the {cap}-turn cap")
|
||||||
|
|
||||||
|
def _circling(self, turns: list[circles.Turn]) -> str | None:
|
||||||
|
"""What the run is going in circles on, if it is (L2): the same
|
||||||
|
calls getting the same back, one correction given again and again,
|
||||||
|
or apologies in a row. Near-identical calls and scattered apologies
|
||||||
|
are only flagged, on the status page: ``git add a.py`` then
|
||||||
|
``b.py`` is progress."""
|
||||||
|
if self._circle_repeats <= 0:
|
||||||
|
return None
|
||||||
|
found = [
|
||||||
|
c
|
||||||
|
for c in circles.find_circles(
|
||||||
|
turns,
|
||||||
|
similar=self._circle_similar,
|
||||||
|
repeats=self._circle_repeats,
|
||||||
|
exact_calls=True,
|
||||||
|
)
|
||||||
|
if c.kind != "repeated apology"
|
||||||
|
]
|
||||||
|
found.append(circles.apologies_in_a_row(turns, self._circle_repeats))
|
||||||
|
for circle in found:
|
||||||
|
if circle is None:
|
||||||
|
continue
|
||||||
|
last = turns[circle.turns[-1]]
|
||||||
|
doing = "; ".join(f"{tool}({args})" for tool, args in last.calls)
|
||||||
|
doing = doing or last.text
|
||||||
|
return f"{circle.detail}; the last: {doing[:300]}"
|
||||||
|
return None
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _unreachable(
|
def _unreachable(
|
||||||
transcript: Transcript, calls: list[ToolCall], index: int, said: str
|
transcript: Transcript, calls: list[ToolCall], index: int, said: str
|
||||||
@@ -309,6 +376,10 @@ class Driver:
|
|||||||
return RunOutcome("unreachable", note=said)
|
return RunOutcome("unreachable", note=said)
|
||||||
|
|
||||||
|
|
||||||
|
def _args(call: ToolCall) -> str:
|
||||||
|
return json.dumps(dict(call.arguments), sort_keys=True, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
def _brief(call: ToolCall, result: str) -> str:
|
def _brief(call: ToolCall, result: str) -> str:
|
||||||
"""A failed command and the end of what it said, for a helper's brief."""
|
"""A failed command and the end of what it said, for a helper's brief."""
|
||||||
command = str(call.arguments.get("command", ""))[:120]
|
command = str(call.arguments.get("command", ""))[:120]
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
"""Runs going in circles (L2, X6), judged from the turns alone."""
|
"""Runs going in circles (L2, X6), judged from the turns alone."""
|
||||||
|
|
||||||
from nisse_core.loop.circles import Turn, find_circles
|
from nisse_core.loop.circles import Turn, apologies_in_a_row, find_circles
|
||||||
|
|
||||||
NUDGE = "Your last reply was prose. Every reply must be a tool call."
|
NUDGE = "Your last reply was prose. Every reply must be a tool call."
|
||||||
|
|
||||||
@@ -71,3 +71,46 @@ def test_the_limits_are_tuning() -> None:
|
|||||||
turns = [comment(1), comment(2)]
|
turns = [comment(1), comment(2)]
|
||||||
assert [c.kind for c in find_circles(turns, repeats=2)] == ["repeated reply"]
|
assert [c.kind for c in find_circles(turns, repeats=2)] == ["repeated reply"]
|
||||||
assert find_circles([comment(1), comment(22), comment(333)], similar=1.0) == []
|
assert find_circles([comment(1), comment(22), comment(333)], similar=1.0) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_alike_replies_that_got_different_results_are_no_circle() -> None:
|
||||||
|
read = [("read_file", '{"path": "src/a.py"}')]
|
||||||
|
turns = [Turn(calls=tuple(read), results=(f"file {n}",)) for n in range(4)]
|
||||||
|
assert find_circles(turns) == []
|
||||||
|
same = [Turn(calls=tuple(read), results=("posted",)) for _ in range(4)]
|
||||||
|
assert [c.kind for c in find_circles(same)] == ["repeated reply"]
|
||||||
|
# The page gives no results: alike replies alone are a circle there.
|
||||||
|
bare = [Turn(calls=tuple(read)) for _ in range(3)]
|
||||||
|
assert [c.kind for c in find_circles(bare)] == ["repeated reply"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_driver_counts_only_exactly_repeated_calls() -> None:
|
||||||
|
adds = [
|
||||||
|
Turn(calls=(("run_command", f'{{"command": "git add {f}.py"}}'),))
|
||||||
|
for f in "abc"
|
||||||
|
]
|
||||||
|
assert [c.kind for c in find_circles(adds)] == ["repeated reply"] # flagged
|
||||||
|
assert find_circles(adds, exact_calls=True) == [] # never ended for it
|
||||||
|
same = [Turn(calls=(("read_file", '{"path": "a"}'),))] * 3
|
||||||
|
assert len(find_circles(same, exact_calls=True)) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_apologies_end_a_run_only_in_a_row() -> None:
|
||||||
|
sorry, work = Turn(text="Sorry, my mistake."), Turn(text="Reading the file.")
|
||||||
|
assert apologies_in_a_row([work, sorry, sorry, sorry]) is not None
|
||||||
|
assert apologies_in_a_row([sorry, work, sorry, sorry]) is None
|
||||||
|
assert apologies_in_a_row([sorry, sorry]) is None
|
||||||
|
# In what it posted, too: an apology comment is an apology.
|
||||||
|
posted = Turn(calls=(("post_comment", '{"body": "Please disregard (1)."}'),))
|
||||||
|
assert apologies_in_a_row([posted] * 3) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_same_call_answered_with_a_count_is_still_a_circle() -> None:
|
||||||
|
note = (("comment", '{"line": 2, "text": "Same point."}'),)
|
||||||
|
turns = [Turn(calls=note, results=(f"noted ({n} in the draft)",)) for n in range(3)]
|
||||||
|
assert len(find_circles(turns, exact_calls=True)) == 1
|
||||||
|
files = [
|
||||||
|
Turn(calls=(("read_file", f'{{"path": "{f}"}}'),), results=(f"{f}: x",))
|
||||||
|
for f in "abc"
|
||||||
|
]
|
||||||
|
assert find_circles(files, exact_calls=True) == []
|
||||||
@@ -5,7 +5,13 @@ import io
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from nisse_core.log.canonical import CanonicalEvent, ToolCall, ToolResult
|
from nisse_core.log.canonical import (
|
||||||
|
AssistantText,
|
||||||
|
CanonicalEvent,
|
||||||
|
Fact,
|
||||||
|
ToolCall,
|
||||||
|
ToolResult,
|
||||||
|
)
|
||||||
from nisse_core.loop.driver import Driver, ToolOutcome
|
from nisse_core.loop.driver import Driver, ToolOutcome
|
||||||
from nisse_core.loop.tools import tool
|
from nisse_core.loop.tools import tool
|
||||||
from nisse_core.oplog import OpLog
|
from nisse_core.oplog import OpLog
|
||||||
@@ -150,3 +156,156 @@ def test_without_due_a_settled_point_leaves_the_run_going(tmp_path: Path) -> Non
|
|||||||
handlers={"note": lambda args: ("noted", True)},
|
handlers={"note": lambda args: ("noted", True)},
|
||||||
)
|
)
|
||||||
assert (ran.tool, ran.arguments) == ("post_review", {"body": "b"})
|
assert (ran.tool, ran.arguments) == ("post_review", {"body": "b"})
|
||||||
|
|
||||||
|
|
||||||
|
# --- runs going in circles (L2, M7.8) ------------------------------------------
|
||||||
|
|
||||||
|
COMMENT = tool("post_comment", "Post a comment.", body="The comment.")
|
||||||
|
|
||||||
|
|
||||||
|
def facts(log: Any) -> list[str]:
|
||||||
|
return [e.event.text for e in log.read() if isinstance(e.event, Fact)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_run_posting_the_same_comment_ends_as_circling(tmp_path: Path) -> None:
|
||||||
|
# Flash-Next on 2026-10-24: the same apology, numbered, about 20 times
|
||||||
|
# until the turn cap.
|
||||||
|
posted: list[str] = []
|
||||||
|
model = Scripted(
|
||||||
|
*[
|
||||||
|
[call("post_comment", body=f"Duplicate, please disregard ({n}).")]
|
||||||
|
for n in range(20)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
log = transcript(tmp_path)
|
||||||
|
ran = Driver(model, "m", max_turns=30).run(
|
||||||
|
log,
|
||||||
|
system="s",
|
||||||
|
tools=[COMMENT, POST],
|
||||||
|
waiting={"post_review"},
|
||||||
|
workspace=None,
|
||||||
|
handlers={
|
||||||
|
"post_comment": lambda a: (posted.append(a["body"]) or "posted", True)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert ran.kind == "circling"
|
||||||
|
assert len(posted) == 3 # ended at the third, not the thirtieth
|
||||||
|
assert ran.note.startswith("3 apologies in a row")
|
||||||
|
assert "Duplicate, please disregard (2)" in ran.note # what it was doing
|
||||||
|
assert facts(log) == [f"The run was ended: it was going in circles ({ran.note})."]
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_invented_tool_told_again_and_again_ends_the_run(tmp_path: Path) -> None:
|
||||||
|
# A different reply each time, but the same correction every time.
|
||||||
|
model = Scripted(
|
||||||
|
*[[call("search_code", query=f"attempt {n} " * n)] for n in range(1, 10)]
|
||||||
|
)
|
||||||
|
ran = Driver(model, "m").run(
|
||||||
|
transcript(tmp_path),
|
||||||
|
system="s",
|
||||||
|
tools=[READ, POST],
|
||||||
|
waiting={"post_review"},
|
||||||
|
workspace=Reader(),
|
||||||
|
)
|
||||||
|
assert ran.kind == "circling"
|
||||||
|
assert ran.note.startswith("told 3 times: no such tool: search_code")
|
||||||
|
|
||||||
|
|
||||||
|
def test_reading_file_after_file_is_progress_not_a_circle(tmp_path: Path) -> None:
|
||||||
|
class Files:
|
||||||
|
def dispatch(self, tool: str, arguments: dict[str, Any]) -> ToolOutcome:
|
||||||
|
# Near-identical contents: still a different file each time.
|
||||||
|
return ToolOutcome(f"{arguments['path']}\n" + "x = 1\n" * 200, True)
|
||||||
|
|
||||||
|
model = Scripted(
|
||||||
|
*[[call("read_file", path=f"src/test_{n}.py")] for n in range(6)],
|
||||||
|
[call("post_review", body="ok")],
|
||||||
|
)
|
||||||
|
ran = Driver(model, "m").run(
|
||||||
|
transcript(tmp_path),
|
||||||
|
system="s",
|
||||||
|
tools=[READ, POST],
|
||||||
|
waiting={"post_review"},
|
||||||
|
workspace=Files(),
|
||||||
|
)
|
||||||
|
assert ran.kind == "wait"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_same_call_getting_the_same_back_ends_the_run(tmp_path: Path) -> None:
|
||||||
|
model = Scripted(*[[call("read_file", path="a.py")] for _ in range(9)])
|
||||||
|
ran = Driver(model, "m").run(
|
||||||
|
transcript(tmp_path),
|
||||||
|
system="s",
|
||||||
|
tools=[READ, POST],
|
||||||
|
waiting={"post_review"},
|
||||||
|
workspace=Reader(),
|
||||||
|
)
|
||||||
|
assert ran.kind == "circling"
|
||||||
|
assert ran.note.startswith("3 near-identical replies in a row")
|
||||||
|
assert 'read_file({"path": "a.py"})' in ran.note
|
||||||
|
|
||||||
|
|
||||||
|
def test_alike_commands_are_progress_not_a_circle(tmp_path: Path) -> None:
|
||||||
|
# git add a.py, b.py, c.py: alike, each answered the same, each its own.
|
||||||
|
class Git:
|
||||||
|
def dispatch(self, tool: str, arguments: dict[str, Any]) -> ToolOutcome:
|
||||||
|
return ToolOutcome("", True)
|
||||||
|
|
||||||
|
model = Scripted(
|
||||||
|
*[[call("run_command", command=f"git add {f}.py")] for f in "abcdef"],
|
||||||
|
[call("post_review", body="ok")],
|
||||||
|
)
|
||||||
|
ran = Driver(model, "m").run(
|
||||||
|
transcript(tmp_path),
|
||||||
|
system="s",
|
||||||
|
tools=[tool("run_command", "Run.", command="It."), POST],
|
||||||
|
waiting={"post_review"},
|
||||||
|
workspace=Git(),
|
||||||
|
)
|
||||||
|
assert ran.kind == "wait"
|
||||||
|
|
||||||
|
|
||||||
|
def test_circles_can_be_turned_off(tmp_path: Path) -> None:
|
||||||
|
model = Scripted(*[[call("post_comment", body="same")] for _ in range(5)])
|
||||||
|
ran = Driver(model, "m", max_turns=5, circle_repeats=0).run(
|
||||||
|
transcript(tmp_path),
|
||||||
|
system="s",
|
||||||
|
tools=[COMMENT, POST],
|
||||||
|
waiting={"post_review"},
|
||||||
|
workspace=None,
|
||||||
|
handlers={"post_comment": lambda a: ("posted", True)},
|
||||||
|
)
|
||||||
|
assert ran.kind == "capped"
|
||||||
|
|
||||||
|
|
||||||
|
def test_repeats_and_likeness_are_the_agents_to_set(tmp_path: Path) -> None:
|
||||||
|
model = Scripted(*[[call("post_comment", body="same")] for _ in range(5)])
|
||||||
|
ran = Driver(model, "m", circle_repeats=5).run(
|
||||||
|
transcript(tmp_path),
|
||||||
|
system="s",
|
||||||
|
tools=[COMMENT, POST],
|
||||||
|
waiting={"post_review"},
|
||||||
|
workspace=None,
|
||||||
|
handlers={"post_comment": lambda a: ("posted", True)},
|
||||||
|
)
|
||||||
|
assert ran.kind == "circling" and ran.note.startswith("5 near-identical")
|
||||||
|
|
||||||
|
|
||||||
|
def test_apologies_spread_over_a_run_dont_end_it(tmp_path: Path) -> None:
|
||||||
|
# Only flagged on the status page: sorry now and then is not a loop.
|
||||||
|
def turn(n: int, sorry: bool) -> list[CanonicalEvent]:
|
||||||
|
said = "Sorry, wrong file." if sorry else "Reading on."
|
||||||
|
return [AssistantText(text=said, model="m"), call("read_file", path=f"f{n}")]
|
||||||
|
|
||||||
|
model = Scripted(
|
||||||
|
*[turn(n, sorry=n % 2 == 0) for n in range(7)],
|
||||||
|
[call("post_review", body="ok")],
|
||||||
|
)
|
||||||
|
ran = Driver(model, "m").run(
|
||||||
|
transcript(tmp_path),
|
||||||
|
system="s",
|
||||||
|
tools=[READ, POST],
|
||||||
|
waiting={"post_review"},
|
||||||
|
workspace=Reader(),
|
||||||
|
)
|
||||||
|
assert ran.kind == "wait"
|
||||||
@@ -100,6 +100,8 @@ tuning:
|
|||||||
driver:
|
driver:
|
||||||
max_turns: 40 # model calls per driver run before it stops as capped
|
max_turns: 40 # model calls per driver run before it stops as capped
|
||||||
failure_limit: 2 # consecutive failed commands before escalating (L9)
|
failure_limit: 2 # consecutive failed commands before escalating (L9)
|
||||||
|
circle_repeats: 3 # a run repeating itself this often ends, escalated (L2); 0: never
|
||||||
|
circle_similar: 0.9 # how alike its repeats must be
|
||||||
preflight: # the toolchain step (D3)
|
preflight: # the toolchain step (D3)
|
||||||
max_turns: 20 # model calls one toolchain step may make
|
max_turns: 20 # model calls one toolchain step may make
|
||||||
command_timeout_s: 900 # one command in it: installs take a while
|
command_timeout_s: 900 # one command in it: installs take a while
|
||||||
|
|||||||
@@ -266,6 +266,8 @@ def run_task(
|
|||||||
model,
|
model,
|
||||||
max_turns=tuning.driver.max_turns,
|
max_turns=tuning.driver.max_turns,
|
||||||
failure_limit=tuning.driver.failure_limit,
|
failure_limit=tuning.driver.failure_limit,
|
||||||
|
circle_repeats=tuning.driver.circle_repeats,
|
||||||
|
circle_similar=tuning.driver.circle_similar,
|
||||||
# As the agent runs it, so a long task measures the same thing.
|
# As the agent runs it, so a long task measures the same thing.
|
||||||
compactor=Compactor(meter, tuning.context.compaction()),
|
compactor=Compactor(meter, tuning.context.compaction()),
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -25,6 +25,7 @@ from nisse_core.config import (
|
|||||||
REFERENCE,
|
REFERENCE,
|
||||||
ConfigError,
|
ConfigError,
|
||||||
VaultConfig,
|
VaultConfig,
|
||||||
|
check_circles,
|
||||||
flatten,
|
flatten,
|
||||||
llm_numbers,
|
llm_numbers,
|
||||||
parse_tuning,
|
parse_tuning,
|
||||||
@@ -107,6 +108,10 @@ GENERAL = "general" # the scope of conversation not about a project (L1)
|
|||||||
class DriverTuning:
|
class DriverTuning:
|
||||||
max_turns: int = 40
|
max_turns: int = 40
|
||||||
failure_limit: int = 2
|
failure_limit: int = 2
|
||||||
|
# A run going in circles ends (L2): this many alike turns in a row, or
|
||||||
|
# one correction given this often; 0 never ends one. And how alike.
|
||||||
|
circle_repeats: int = field(default=3, metadata={"zero": True})
|
||||||
|
circle_similar: float = 0.9
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -199,6 +204,12 @@ def _parse_tuning(raw: Any, errors: list[str]) -> Tuning:
|
|||||||
if tuning.context.safety_margin >= 1:
|
if tuning.context.safety_margin >= 1:
|
||||||
errors.append("tuning.context.safety_margin: must be less than 1")
|
errors.append("tuning.context.safety_margin: must be less than 1")
|
||||||
check_tuning(tuning.context.compaction(), "tuning.context", errors)
|
check_tuning(tuning.context.compaction(), "tuning.context", errors)
|
||||||
|
check_circles(
|
||||||
|
tuning.driver.circle_repeats,
|
||||||
|
tuning.driver.circle_similar,
|
||||||
|
"tuning.driver",
|
||||||
|
errors,
|
||||||
|
)
|
||||||
return tuning
|
return tuning
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -2378,11 +2378,12 @@ class LoopAgent:
|
|||||||
|
|
||||||
def _help_when_stuck(self, item: WorkItem, loop: Loop, outcome: RunOutcome) -> bool:
|
def _help_when_stuck(self, item: WorkItem, loop: Loop, outcome: RunOutcome) -> bool:
|
||||||
"""L9's observable triggers: the driver stuck (commands failing in a
|
"""L9's observable triggers: the driver stuck (commands failing in a
|
||||||
row) or out of turns mid-milestone. A helper is called in, once per
|
row), going in circles (L2), or out of turns mid-milestone. A helper
|
||||||
milestone; True when it finished and the driver goes on."""
|
is called in, once per milestone; True when it finished and the
|
||||||
|
driver goes on."""
|
||||||
if (
|
if (
|
||||||
self._helpers is None
|
self._helpers is None
|
||||||
or outcome.kind not in ("stuck", "capped")
|
or outcome.kind not in ("stuck", "circling", "capped")
|
||||||
or not loop.branch
|
or not loop.branch
|
||||||
or loop.helped_at == loop.milestone
|
or loop.helped_at == loop.milestone
|
||||||
):
|
):
|
||||||
|
|||||||
@@ -292,6 +292,8 @@ def build_loop_agent(
|
|||||||
max_turns=config.tuning.driver.max_turns,
|
max_turns=config.tuning.driver.max_turns,
|
||||||
cancel=cancel,
|
cancel=cancel,
|
||||||
failure_limit=config.tuning.driver.failure_limit,
|
failure_limit=config.tuning.driver.failure_limit,
|
||||||
|
circle_repeats=config.tuning.driver.circle_repeats,
|
||||||
|
circle_similar=config.tuning.driver.circle_similar,
|
||||||
compactor=Compactor(
|
compactor=Compactor(
|
||||||
normalized, config.tuning.context.compaction(), oplog=oplog
|
normalized, config.tuning.context.compaction(), oplog=oplog
|
||||||
),
|
),
|
||||||
|
|||||||
@@ -182,6 +182,23 @@ def test_tuning_is_read() -> None:
|
|||||||
assert (tuning.build_vm.retry_s, tuning.build_vm.settle_s) == (60.0, 180.0)
|
assert (tuning.build_vm.retry_s, tuning.build_vm.settle_s) == (60.0, 180.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_circles_are_tuned_and_checked() -> None:
|
||||||
|
# L2, M7.8: when a driver run going in circles ends.
|
||||||
|
assert (
|
||||||
|
build().tuning.driver.circle_repeats,
|
||||||
|
build().tuning.driver.circle_similar,
|
||||||
|
) == (3, 0.9)
|
||||||
|
tuned = build(tuning={"driver": {"circle_repeats": 0, "circle_similar": 0.8}})
|
||||||
|
assert (tuned.tuning.driver.circle_repeats, tuned.tuning.driver.circle_similar) == (
|
||||||
|
0,
|
||||||
|
0.8,
|
||||||
|
)
|
||||||
|
with pytest.raises(ConfigError, match=r"circle_repeats: must be 0 \(off\) or 2"):
|
||||||
|
build(tuning={"driver": {"circle_repeats": 1}})
|
||||||
|
with pytest.raises(ConfigError, match="circle_similar: must be at most 1"):
|
||||||
|
build(tuning={"driver": {"circle_similar": 2}})
|
||||||
|
|
||||||
|
|
||||||
def test_a_partly_given_tuning_section_keeps_the_other_defaults() -> None:
|
def test_a_partly_given_tuning_section_keeps_the_other_defaults() -> None:
|
||||||
tuning = build(tuning={"driver": {"max_turns": 40}}).tuning
|
tuning = build(tuning={"driver": {"max_turns": 40}}).tuning
|
||||||
assert tuning.driver.failure_limit == 2
|
assert tuning.driver.failure_limit == 2
|
||||||
|
|||||||
@@ -386,7 +386,8 @@ def test_an_invented_tool_is_answered_not_run(tmp_path: Path) -> None:
|
|||||||
|
|
||||||
def test_the_turn_cap_ends_a_run(tmp_path: Path) -> None:
|
def test_the_turn_cap_ends_a_run(tmp_path: Path) -> None:
|
||||||
model = ScriptedModel(*[[call("read_file", path="a")]] * 3)
|
model = ScriptedModel(*[[call("read_file", path="a")]] * 3)
|
||||||
outcome = Driver(model, "m", max_turns=3).run(
|
# The same call three times is a circle too (M7.8): here, only the cap.
|
||||||
|
outcome = Driver(model, "m", max_turns=3, circle_repeats=0).run(
|
||||||
driver_log(tmp_path),
|
driver_log(tmp_path),
|
||||||
system="s",
|
system="s",
|
||||||
tools=_tools("read_file"),
|
tools=_tools("read_file"),
|
||||||
|
|||||||
@@ -249,6 +249,49 @@ def test_stuck_again_in_the_same_milestone_asks_the_operator(
|
|||||||
assert len(helper.calls) == 2 # one delegation: its two turns
|
assert len(helper.calls) == 2 # one delegation: its two turns
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_driver_going_in_circles_gets_a_helper_too(world: dict[str, Any]) -> None:
|
||||||
|
# L2, M7.8: the same command, the same output, turn after turn.
|
||||||
|
helper = Spy(
|
||||||
|
[call("write_file", path="fixed.py", content="F = 1\n")],
|
||||||
|
[hand_back()],
|
||||||
|
)
|
||||||
|
model = ScriptedModel(
|
||||||
|
RESTATE,
|
||||||
|
[call("start_milestone", name="words")],
|
||||||
|
*[[call("run_command", command="true")] for _ in range(3)],
|
||||||
|
[call("finish_milestone", summary="fixed with help", final="yes")],
|
||||||
|
)
|
||||||
|
agent, transport = loop_agent(world, model, helpers=helpers(helper))
|
||||||
|
agent.intake()
|
||||||
|
say(transport, world, "go")
|
||||||
|
|
||||||
|
assert any("so a helper is taking over" in m.text for m in transport.outbound)
|
||||||
|
assert "near-identical replies in a row" in str(helper.calls) # in the brief
|
||||||
|
assert loop_of(item(world)).helped_at == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_circling_again_in_the_same_milestone_asks_the_operator(
|
||||||
|
world: dict[str, Any],
|
||||||
|
) -> None:
|
||||||
|
helper = Spy(
|
||||||
|
[call("write_file", path="tried.py", content="T = 1\n")],
|
||||||
|
[hand_back()],
|
||||||
|
)
|
||||||
|
circling = [[call("run_command", command="true")] for _ in range(3)]
|
||||||
|
model = ScriptedModel(
|
||||||
|
RESTATE,
|
||||||
|
[call("start_milestone", name="words")],
|
||||||
|
*circling,
|
||||||
|
*circling, # after the helper: round again
|
||||||
|
)
|
||||||
|
agent, transport = loop_agent(world, model, helpers=helpers(helper))
|
||||||
|
agent.intake()
|
||||||
|
say(transport, world, "go")
|
||||||
|
loop = loop_of(item(world))
|
||||||
|
assert loop.wait == ANSWER
|
||||||
|
assert "I stopped short: 3 near-identical replies in a row" in loop.note
|
||||||
|
|
||||||
|
|
||||||
def test_ask_expert_answers_within_its_budget(world: dict[str, Any]) -> None:
|
def test_ask_expert_answers_within_its_budget(world: dict[str, Any]) -> None:
|
||||||
answer = {"answer": "use a dict", "confidence": "high", "caveats": ""}
|
answer = {"answer": "use a dict", "confidence": "high", "caveats": ""}
|
||||||
expert = Spy(
|
expert = Spy(
|
||||||
|
|||||||
@@ -139,8 +139,10 @@ def test_a_pass_that_doesnt_finish_never_holds_up_the_close(
|
|||||||
|
|
||||||
|
|
||||||
def test_the_pass_is_bounded(world: dict[str, Any], memory: Memory) -> None:
|
def test_the_pass_is_bounded(world: dict[str, Any], memory: Memory) -> None:
|
||||||
looking = [call("list_notes", folder="")]
|
# A different folder each turn: the same call three times would end the
|
||||||
model = Spy(RESTATE, *[looking] * 3)
|
# pass as going in circles (M7.8), not at the cap.
|
||||||
|
looking = [[call("list_notes", folder=f)] for f in ("", "a", "b")]
|
||||||
|
model = Spy(RESTATE, *looking)
|
||||||
loop, transport = agent(world, model, memory, memory_turns=3)
|
loop, transport = agent(world, model, memory, memory_turns=3)
|
||||||
loop.intake()
|
loop.intake()
|
||||||
say(transport, world, "abandon")
|
say(transport, world, "abandon")
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ gitea:
|
|||||||
|
|
||||||
# tuning: # docs/Tuning-Guide.md
|
# tuning: # docs/Tuning-Guide.md
|
||||||
# review: {poll_s: 60, max_turns: 40, diff_chars: 60000, attempts: 2,
|
# review: {poll_s: 60, max_turns: 40, diff_chars: 60000, attempts: 2,
|
||||||
# ci_wait_s: 3600, ci_grace_s: 120, max_rounds: 4}
|
# ci_wait_s: 3600, ci_grace_s: 120, max_rounds: 4,
|
||||||
|
# circle_repeats: 3, circle_similar: 0.9}
|
||||||
# expert: {consult_budget: 3}
|
# expert: {consult_budget: 3}
|
||||||
# context: {keep_recent_turns: 6, safety_margin: 0.1}
|
# context: {keep_recent_turns: 6, safety_margin: 0.1}
|
||||||
@@ -138,6 +138,8 @@ def build(
|
|||||||
config.roles.resolve("driver"),
|
config.roles.resolve("driver"),
|
||||||
cancel=cancel,
|
cancel=cancel,
|
||||||
compactor=Compactor(model, tuning.context.compaction(), oplog=oplog),
|
compactor=Compactor(model, tuning.context.compaction(), oplog=oplog),
|
||||||
|
circle_repeats=tuning.review.circle_repeats,
|
||||||
|
circle_similar=tuning.review.circle_similar,
|
||||||
),
|
),
|
||||||
account=account,
|
account=account,
|
||||||
gitea_url=config.gitea_url,
|
gitea_url=config.gitea_url,
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ from typing import Any
|
|||||||
from nisse_core.config import (
|
from nisse_core.config import (
|
||||||
ConfigError,
|
ConfigError,
|
||||||
VaultConfig,
|
VaultConfig,
|
||||||
|
check_circles,
|
||||||
flatten,
|
flatten,
|
||||||
llm_numbers,
|
llm_numbers,
|
||||||
parse_tuning,
|
parse_tuning,
|
||||||
@@ -64,6 +65,10 @@ class ReviewTuning:
|
|||||||
ci_grace_s: float = field(default=120.0, metadata={"zero": True})
|
ci_grace_s: float = field(default=120.0, metadata={"zero": True})
|
||||||
# Reviews and reply sets before it sums up for the operator (R2).
|
# Reviews and reply sets before it sums up for the operator (R2).
|
||||||
max_rounds: int = 4
|
max_rounds: int = 4
|
||||||
|
# A run going in circles ends (L2): this many alike turns in a row, or
|
||||||
|
# one correction given this often; 0 never ends one. And how alike.
|
||||||
|
circle_repeats: int = field(default=3, metadata={"zero": True})
|
||||||
|
circle_similar: float = 0.9
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -136,6 +141,12 @@ class ReviewConfig:
|
|||||||
if tuning.context.safety_margin >= 1:
|
if tuning.context.safety_margin >= 1:
|
||||||
errors.append("tuning.context.safety_margin: must be less than 1")
|
errors.append("tuning.context.safety_margin: must be less than 1")
|
||||||
check_tuning(tuning.context.compaction(), "tuning.context", errors)
|
check_tuning(tuning.context.compaction(), "tuning.context", errors)
|
||||||
|
check_circles(
|
||||||
|
tuning.review.circle_repeats,
|
||||||
|
tuning.review.circle_similar,
|
||||||
|
"tuning.review",
|
||||||
|
errors,
|
||||||
|
)
|
||||||
vault = parse_vault(tree.get("vault"), errors)
|
vault = parse_vault(tree.get("vault"), errors)
|
||||||
flat = flatten({k: v for k, v in tree.items() if k != "vault"})
|
flat = flatten({k: v for k, v in tree.items() if k != "vault"})
|
||||||
flat.pop("tuning", None)
|
flat.pop("tuning", None)
|
||||||
|
|||||||
@@ -492,6 +492,29 @@ def test_the_ci_waits_may_be_zero(tmp_path: Path) -> None:
|
|||||||
}
|
}
|
||||||
review = ReviewConfig.from_tree(tree, {}).tuning.review
|
review = ReviewConfig.from_tree(tree, {}).tuning.review
|
||||||
assert (review.ci_wait_s, review.ci_grace_s) == (0, 0)
|
assert (review.ci_wait_s, review.ci_grace_s) == (0, 0)
|
||||||
|
assert (review.circle_repeats, review.circle_similar) == (3, 0.9)
|
||||||
|
|
||||||
|
|
||||||
|
def test_circles_are_tuned_in_the_config(tmp_path: Path) -> None:
|
||||||
|
from nisse_core.config import ConfigError
|
||||||
|
from nisse_review.config import ReviewConfig
|
||||||
|
|
||||||
|
tree = {
|
||||||
|
"state_db": str(tmp_path / "s.db"),
|
||||||
|
"llm": {"base_url": "http://llm", "token": "k"},
|
||||||
|
"roles": {"reviewer": "local", "expert": "frontier"},
|
||||||
|
"gitea": {"url": "https://g", "token": "t", "operators": ["don"]},
|
||||||
|
}
|
||||||
|
off = {**tree, "tuning": {"review": {"circle_repeats": 0}}}
|
||||||
|
assert ReviewConfig.from_tree(off, {}).tuning.review.circle_repeats == 0
|
||||||
|
for bad, message in (
|
||||||
|
({"circle_repeats": 1}, "circle_repeats: must be 0 (off) or 2 or more"),
|
||||||
|
({"circle_similar": 1.5}, "circle_similar: must be at most 1"),
|
||||||
|
):
|
||||||
|
with pytest.raises(
|
||||||
|
ConfigError, match=message.replace("(", r"\(").replace(")", r"\)")
|
||||||
|
):
|
||||||
|
ReviewConfig.from_tree({**tree, "tuning": {"review": bad}}, {})
|
||||||
|
|
||||||
|
|
||||||
def test_compaction_is_tuned_in_the_config(tmp_path: Path) -> None:
|
def test_compaction_is_tuned_in_the_config(tmp_path: Path) -> None:
|
||||||
@@ -627,6 +650,20 @@ def test_out_of_turns_with_nothing_drafted_is_a_failed_run(tmp_path: Path) -> No
|
|||||||
assert (item(w).position, pr_of(item(w)).attempts) == (READY, 1)
|
assert (item(w).position, pr_of(item(w)).attempts) == (READY, 1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_run_going_in_circles_fails_and_its_repeats_are_not_posted(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
# L2, M7.8: the same comment drafted again and again ends the run as
|
||||||
|
# circling, a failed run; its drafts are the repeats, so none is posted.
|
||||||
|
again = [call("comment", path="count.py", line="2", text="Same point.")]
|
||||||
|
model = Scripted(again, again, again, again, again)
|
||||||
|
w = world(tmp_path, model, attempts=2)
|
||||||
|
w["scheduler"].round()
|
||||||
|
assert w["gitea"].posted == []
|
||||||
|
assert (item(w).position, pr_of(item(w)).attempts) == (READY, 1)
|
||||||
|
assert len(model.seen) == 3 # ended at the third, not at the cap
|
||||||
|
|
||||||
|
|
||||||
def test_code_git_cannot_give_fails_the_run(tmp_path: Path) -> None:
|
def test_code_git_cannot_give_fails_the_run(tmp_path: Path) -> None:
|
||||||
from nisse_review.clones import CloneError
|
from nisse_review.clones import CloneError
|
||||||
|
|
||||||
|
|||||||
@@ -136,6 +136,7 @@ def circling(root: Path) -> Source:
|
|||||||
Driver(
|
Driver(
|
||||||
client(Server(*answers, reply(call=("finish", {"summary": "x"}))), recorder),
|
client(Server(*answers, reply(call=("finish", {"summary": "x"}))), recorder),
|
||||||
"local",
|
"local",
|
||||||
|
circle_repeats=0, # as it ran then: before M7.8 ended such runs
|
||||||
).run(
|
).run(
|
||||||
store.transcript("pr-7", "driver"),
|
store.transcript("pr-7", "driver"),
|
||||||
system="You review.",
|
system="You review.",
|
||||||
|
|||||||
Reference in new issue
Block a user