Discover a model's vendor from LiteLLM instead of its name
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 47s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 47s
Route selection sniffed model names: anything starting "claude-" or "anthropic/" took the vendor-native passthrough, everything else took the normalized route. That works only while an alias names its vendor. A semantic role alias like "remote-architecture" is meant to point at Anthropic today and somewhere else tomorrow, and the name says nothing either way -- so it would have taken the normalized route regardless of what was behind it, silently losing the fidelity A13 wants for delegations. LiteLLM already knows the answer. /model/info reports each alias's backing model, provider, and declared tool support, so the agent asks at startup and routes on the provider it discovers. That keeps vendor knowledge where A5 already put it -- in the proxy -- rather than duplicating it as name patterns here. There is deliberately no name-based fallback: PASSTHROUGH_PREFIXES and has_native_adapter are gone, and select_client now requires a catalog. A second heuristic could only ever disagree with the proxy. An unknown model raises instead of guessing a route. Misconfiguration is therefore loud and early. build_agent validates every role against the catalog before it clones anything, naming all the bad ones at once. Discovery is per key -- /model/info returns only what the presented token may use -- so the same check catches a role pointed at a real model this key cannot reach. An unreachable proxy is reported separately, since an outage and a typo call for different responses from the operator. Two things now get recorded rather than inferred: the resolved model is pinned alongside the role mapping at task start, for the reason A11 already pins roles, and DelegationStarted carries backing_model so A16 eval history is not averaged across whatever an alias happened to point at. A model LiteLLM declares tool-less warns at startup rather than refusing; A15 expects weak local models to be accommodated, and they are observed driving tool loops anyway. Verified live: all four aliases the production key can see route correctly through discovery, and each chosen route works. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
231b406393
commit
e38adf40ca
10 files changed
+574
-76
No files matched your search
@@ -21,9 +21,7 @@ from nisse_dev.llm import (
|
||||
RoleMap,
|
||||
TruncatedResponse,
|
||||
UpstreamUnavailable,
|
||||
has_native_adapter,
|
||||
max_output_tokens,
|
||||
select_client,
|
||||
)
|
||||
|
||||
EVENTS = [StoredEvent(id=1, ts="t", event=UserMessage(text="hi", source="m:1"))]
|
||||
@@ -218,31 +216,6 @@ def test_load_env_file_missing_is_noop(tmp_path: Path) -> None:
|
||||
assert env == {}
|
||||
|
||||
|
||||
# --- route selection (A5 vendor fluidity) --------------------------------
|
||||
|
||||
|
||||
def test_anthropic_models_take_the_native_route() -> None:
|
||||
normalized, passthrough = object(), object()
|
||||
for model in ("claude-haiku-4-5", "claude-opus-5", "anthropic/claude-sonnet-5"):
|
||||
assert select_client(model, normalized, passthrough) is passthrough
|
||||
|
||||
|
||||
def test_other_vendors_take_the_normalized_route() -> None:
|
||||
# A13: provisionally accept LiteLLM's normalization for a vendor whose
|
||||
# renderer adapter does not exist yet, rather than sending it request
|
||||
# bodies shaped for someone else.
|
||||
normalized, passthrough = object(), object()
|
||||
for model in ("gpt-6-astra", "local-coder", "local-fast", "llama3.2"):
|
||||
assert select_client(model, normalized, passthrough) is normalized
|
||||
|
||||
|
||||
def test_native_adapter_detection_is_explicit() -> None:
|
||||
assert has_native_adapter("claude-opus-5")
|
||||
assert not has_native_adapter("gpt-6-astra")
|
||||
# A model merely mentioning the vendor is not the vendor.
|
||||
assert not has_native_adapter("my-claude-clone")
|
||||
|
||||
|
||||
# --- truncation (silent until measured against Opus) ---------------------
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user