Don't assert the exact word in the LiteLLM smoke test #67

2026-09-29T17:22:25Z

dev-agent commented

2026-09-29 17:22:25 +00:00

test_executor_model_completes_via_litellm checks the plumbing: a request reaches the executor model through LiteLLM and text comes back. It also asserted that the model obeyed "Reply with exactly one word: pong". On #66's CI, Haiku answered "ping" and the job failed; a re-run passed.

What the model chooses to say isn't what this test is for, so it now asserts non-empty text. Other live tests cover model behaviour, e.g. the capability matrix and the planning and milestone tests.

Passes locally against the live proxy; lint and format clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

`test_executor_model_completes_via_litellm` checks the plumbing: a request reaches the executor model through LiteLLM and text comes back. It also asserted that the model obeyed "Reply with exactly one word: pong". On #66's CI, Haiku answered "ping" and the job failed; a re-run passed. What the model chooses to say isn't what this test is for, so it now asserts non-empty text. Other live tests cover model behaviour, e.g. the capability matrix and the planning and milestone tests. Passes locally against the live proxy; lint and format clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dev-agent added 1 commit 2026-09-29 17:22:25 +00:00

Don't assert the exact word in the LiteLLM smoke test

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m24s

Details

1ab5d86a88

The test checks that a request reaches the executor model through
LiteLLM and text comes back. It also asserted the model obeyed "Reply
with exactly one word: pong", which Haiku once answered with "ping" and
failed CI on #66. Assert non-empty text instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dongar approved these changes

2026-09-29 17:22:49 +00:00

dongar merged commit ab932f2ef4 into main

2026-09-29 17:24:44 +00:00

dongar deleted branch smoke-test-no-exact-word

2026-09-29 17:24:44 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#67