Merge pull request 'Expert and integration roles on claude-haiku-5-5' (#134) from haiku-5-5 into main
This commit was merged in pull request #134.
This commit is contained in:
commit
33f9a75bac
4 files changed
+10
-9
No files matched your search
@@ -52,8 +52,8 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
|
|||||||
- **Local inference:** llama-swap on an RTX 4090 host (320 W cap). The benchmark
|
- **Local inference:** llama-swap on an RTX 4090 host (320 W cap). The benchmark
|
||||||
(PR #55) chose `qwen3.8-27b` for local work.
|
(PR #55) chose `qwen3.8-27b` for local work.
|
||||||
- **Models by role today:** driver and helper `claude-opus-5-5`, expert
|
- **Models by role today:** driver and helper `claude-opus-5-5`, expert
|
||||||
`claude-haiku-4-5`. `qwen3.8-27b` has had no role since the executor's went
|
`claude-haiku-5-5`, classifier `qwen3.8-27b` (M3.2); the driver benchmark
|
||||||
(M2.8); the driver benchmark measures where it fits.
|
measures where the local model fits beyond that.
|
||||||
- **Bring-up** runs on this development VM (`scripts/bringup.sh`), against test
|
- **Bring-up** runs on this development VM (`scripts/bringup.sh`), against test
|
||||||
resources: a test repository, the test Matrix room, a budget-capped LiteLLM key;
|
resources: a test repository, the test Matrix room, a budget-capped LiteLLM key;
|
||||||
configuration in `/root/nisse-e2e/`.
|
configuration in `/root/nisse-e2e/`.
|
||||||
|
|||||||
@@ -48,7 +48,7 @@ roles: # L9: role -> LiteLLM model alias
|
|||||||
# Each work item pins the mapping when it opens.
|
# Each work item pins the mapping when it opens.
|
||||||
driver: claude-opus-5-5 # drives the loop: talks, restates, works
|
driver: claude-opus-5-5 # drives the loop: talks, restates, works
|
||||||
helper: claude-opus-5-5 # takes over what the driver got stuck on
|
helper: claude-opus-5-5 # takes over what the driver got stuck on
|
||||||
expert: claude-haiku-4-5 # one question at a time, no tools: cheap is enough
|
expert: claude-haiku-5-5 # one question at a time, no tools: cheap is enough
|
||||||
# Reads go-aheads and judges actions the fast path doesn't know (L4); unset,
|
# Reads go-aheads and judges actions the fast path doesn't know (L4); unset,
|
||||||
# the expert's. It never sees the transcript, so the driver's model is fine.
|
# the expert's. It never sees the transcript, so the driver's model is fine.
|
||||||
classifier: qwen3.8-27b
|
classifier: qwen3.8-27b
|
||||||
@@ -61,10 +61,11 @@ profiles:
|
|||||||
tools: reliable (37/40 on the executor benchmark, PR #55)
|
tools: reliable (37/40 on the executor benchmark, PR #55)
|
||||||
strengths: careful multi-step edits that follow a clear brief
|
strengths: careful multi-step edits that follow a clear brief
|
||||||
weaknesses: speed (minutes per hard task) and open-ended design
|
weaknesses: speed (minutes per hard task) and open-ended design
|
||||||
claude-haiku-4-5:
|
claude-haiku-5-5:
|
||||||
tools: reliable
|
tools: reliable
|
||||||
strengths: quick, well-scoped changes and short answers
|
strengths: quick, well-scoped changes and short answers
|
||||||
weaknesses: long multi-file work (15/20 on the harder benchmark tasks)
|
# Measured on its predecessor, claude-haiku-4-5; not benchmarked yet.
|
||||||
|
weaknesses: long multi-file work (15/20 on the harder benchmark tasks, as 4.5)
|
||||||
claude-opus-5-5:
|
claude-opus-5-5:
|
||||||
tools: reliable
|
tools: reliable
|
||||||
strengths: design, debugging, and work other models got stuck on
|
strengths: design, debugging, and work other models got stuck on
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ from nisse_dev.gitea import GiteaClient, GiteaError
|
|||||||
from nisse_dev.llm import RoleMap
|
from nisse_dev.llm import RoleMap
|
||||||
|
|
||||||
# The end-to-end tests run every role on a local model, which costs nothing
|
# The end-to-end tests run every role on a local model, which costs nothing
|
||||||
# per call; NISSE_E2E_MODEL overrides it (e.g. claude-haiku-4-5 for a quick,
|
# per call; NISSE_E2E_MODEL overrides it (e.g. claude-haiku-5-5 for a quick,
|
||||||
# paid run). The no-thinking alias (LiteLLM: chat_template_kwargs
|
# paid run). The no-thinking alias (LiteLLM: chat_template_kwargs
|
||||||
# enable_thinking false): with thinking, some answers ran to 6-8k tokens,
|
# enable_thinking false): with thinking, some answers ran to 6-8k tokens,
|
||||||
# minutes each, and stretched a test from 3.5 to 20 minutes.
|
# minutes each, and stretched a test from 3.5 to 20 minutes.
|
||||||
|
|||||||
@@ -36,9 +36,9 @@ llm:
|
|||||||
# not local-model competence (measured separately by test_model_capability),
|
# not local-model competence (measured separately by test_model_capability),
|
||||||
# so every role uses a model that holds the tool-call contract.
|
# so every role uses a model that holds the tool-call contract.
|
||||||
roles:
|
roles:
|
||||||
driver: claude-haiku-4-5
|
driver: claude-haiku-5-5
|
||||||
helper: claude-haiku-4-5
|
helper: claude-haiku-5-5
|
||||||
expert: claude-haiku-4-5
|
expert: claude-haiku-5-5
|
||||||
|
|
||||||
matrix:
|
matrix:
|
||||||
homeserver: https://matrix.dongar.net
|
homeserver: https://matrix.dongar.net
|
||||||
|
|||||||
Reference in new issue
Block a user