dongarandClaude Opus 5.5
37965aefca
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 2m17s
The first live run of the new loop's e2e test found that a fact recorded while start_milestone was in flight landed between the tool call and its result; both API formats reject that, so every later request in the run failed. Views now keep each result right after its call. Scripted test models check the APIs' ordering rules, so this can't pass unit tests again. The e2e tests no longer depend on the sandbox's history: each run asks for a uniquely named function and works against a base branch of its own, deleted afterwards. The review step asks for a change that can't already be there. The scripted operator gives up after five fruitless answers, and each run keeps its state store for inspection. The driver's prompt says review fixes stay in the current milestone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
59 lines
2.4 KiB
Python
59 lines
2.4 KiB
Python
"""What a real model API insists on, for scripted test models.
|
|
|
|
Both formats the agent sends require a tool call's result to come right
|
|
after it: in Anthropic's, the next message is the user's and starts with a
|
|
tool_result for every tool_use, before any text; in OpenAI's, the assistant
|
|
message with tool_calls is followed directly by a tool message for each. A
|
|
scripted model that doesn't check this lets a broken transcript pass the
|
|
unit tests and be rejected live; this makes it fail here.
|
|
"""
|
|
|
|
from typing import Any
|
|
|
|
from nisse_dev.render import OpenAIChatRenderer
|
|
from nisse_dev.render_anthropic import AnthropicMessagesRenderer
|
|
|
|
|
|
def assert_renderable(events: list[Any]) -> None:
|
|
_anthropic(AnthropicMessagesRenderer().render(list(events), None, "m")["messages"])
|
|
_openai(OpenAIChatRenderer().render(list(events), None, "m")["messages"])
|
|
|
|
|
|
def _anthropic(messages: list[dict[str, Any]]) -> None:
|
|
for index, message in enumerate(messages):
|
|
if message["role"] != "assistant":
|
|
continue
|
|
used = [b["id"] for b in message["content"] if b.get("type") == "tool_use"]
|
|
if not used:
|
|
continue
|
|
following = messages[index + 1] if index + 1 < len(messages) else None
|
|
assert following is not None and following["role"] == "user", (
|
|
f"anthropic messages.{index}: tool_use with no user message after it"
|
|
)
|
|
blocks = following["content"]
|
|
leading = []
|
|
for block in blocks:
|
|
if block.get("type") != "tool_result":
|
|
break
|
|
leading.append(block["tool_use_id"])
|
|
assert set(used) <= set(leading), (
|
|
f"anthropic messages.{index}: tool_use ids {sorted(set(used) - set(leading))} "
|
|
"lack a tool_result at the start of the next message"
|
|
)
|
|
|
|
|
|
def _openai(messages: list[dict[str, Any]]) -> None:
|
|
for index, message in enumerate(messages):
|
|
calls = [c["id"] for c in message.get("tool_calls") or []]
|
|
if not calls:
|
|
continue
|
|
answered = []
|
|
for after in messages[index + 1 :]:
|
|
if after["role"] != "tool":
|
|
break
|
|
answered.append(after["tool_call_id"])
|
|
assert set(calls) <= set(answered), (
|
|
f"openai messages.{index}: tool_calls {sorted(set(calls) - set(answered))} "
|
|
"are not answered by the tool messages right after them"
|
|
)
|