Discover a model's vendor from LiteLLM instead of its name #35

2026-09-23T01:30:25Z

dev-agent commented

2026-09-23 01:30:25 +00:00

Route selection sniffed model names — claude-/anthropic/ took the vendor-native passthrough, everything else took the normalized route. That only works while an alias names its vendor. A semantic role alias like remote-architecture is meant to be repointed at any time, so it would have taken the normalized route no matter what was behind it, quietly losing the fidelity A13 wants for delegations.

LiteLLM already knows. /model/info reports each alias's backing model, provider, and declared tool support, so the agent asks at startup and routes on the discovered provider — keeping vendor knowledge in the proxy where A5 put it.

No name-based fallback. PASSTHROUGH_PREFIXES and has_native_adapter are deleted; select_client requires a catalog. A second heuristic could only ever disagree with the proxy, and an unknown model raises rather than guessing.

Misconfiguration is loud and early. build_agent validates every role before cloning anything, naming all the bad ones at once. Discovery is per key — /model/info returns only what the presented token may use — so the same check catches a role pointed at a real model this key cannot reach. An unreachable proxy gets a distinct error: an outage and a typo call for different responses.

Now recorded, not inferred: the resolved model is pinned alongside the role mapping at task start (A11's reasoning applied to the route), and DelegationStarted carries backing_model so A16 eval history isn't averaged across whatever an alias pointed at.

A model LiteLLM declares tool-less warns at startup rather than refusing — A15 expects weak local models to be accommodated, and they drive tool loops anyway.

Verification

315 unit + 17 integration tests pass. Live against the production key, all four visible aliases routed correctly through discovery and each chosen route worked.

Two findings from building this, both worth knowing independently of the code:

/model/info is scoped per key. The unit-test key sees claude-haiku-4-5, llama3.2; the production key sees claude-opus-5, gpt-6-astra, local-coder, local-fast. The production key currently cannot see Haiku or Sonnet.
local-coder and local-fast are different models on one llama-swap server (qwen3-14b and qwen3-8b-awq). Pooled over three runs on the 4-tool executor schema, local-fast held the contract 20/20 and local-coder 12/20 (Fisher exact, two-tailed p = 0.0033) — so the smaller model is the better executor candidate despite the name.

🤖 Generated with Claude Code

Route selection sniffed model names — `claude-`/`anthropic/` took the vendor-native passthrough, everything else took the normalized route. That only works while an alias names its vendor. A semantic role alias like `remote-architecture` is meant to be repointed at any time, so it would have taken the normalized route no matter what was behind it, quietly losing the fidelity A13 wants for delegations. LiteLLM already knows. `/model/info` reports each alias's backing model, provider, and declared tool support, so the agent asks at startup and routes on the discovered provider — keeping vendor knowledge in the proxy where A5 put it. **No name-based fallback.** `PASSTHROUGH_PREFIXES` and `has_native_adapter` are deleted; `select_client` requires a catalog. A second heuristic could only ever disagree with the proxy, and an unknown model raises rather than guessing. **Misconfiguration is loud and early.** `build_agent` validates every role before cloning anything, naming all the bad ones at once. Discovery is per key — `/model/info` returns only what the presented token may use — so the same check catches a role pointed at a real model this key cannot reach. An unreachable proxy gets a distinct error: an outage and a typo call for different responses. **Now recorded, not inferred:** the resolved model is pinned alongside the role mapping at task start (A11's reasoning applied to the route), and `DelegationStarted` carries `backing_model` so A16 eval history isn't averaged across whatever an alias pointed at. A model LiteLLM declares tool-less warns at startup rather than refusing — A15 expects weak local models to be accommodated, and they drive tool loops anyway. ### Verification 315 unit + 17 integration tests pass. Live against the production key, all four visible aliases routed correctly through discovery and each chosen route worked. Two findings from building this, both worth knowing independently of the code: - `/model/info` is **scoped per key**. The unit-test key sees `claude-haiku-4-5, llama3.2`; the production key sees `claude-opus-5, gpt-6-astra, local-coder, local-fast`. The production key currently cannot see Haiku or Sonnet. - `local-coder` and `local-fast` are **different models** on one llama-swap server (`qwen3-14b` and `qwen3-8b-awq`). Pooled over three runs on the 4-tool executor schema, `local-fast` held the contract 20/20 and `local-coder` 12/20 (Fisher exact, two-tailed p = 0.0033) — so the smaller model is the better executor candidate despite the name. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dongar changed target branch from model-routing to main

2026-09-23 01:47:36 +00:00

dongar approved these changes

2026-09-23 01:47:47 +00:00

dongar added 1 commit 2026-09-23 01:47:59 +00:00

Discover a model's vendor from LiteLLM instead of its name

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 47s

Details

e38adf40ca

Route selection sniffed model names: anything starting "claude-" or
"anthropic/" took the vendor-native passthrough, everything else took the
normalized route. That works only while an alias names its vendor. A
semantic role alias like "remote-architecture" is meant to point at
Anthropic today and somewhere else tomorrow, and the name says nothing
either way -- so it would have taken the normalized route regardless of
what was behind it, silently losing the fidelity A13 wants for delegations.

LiteLLM already knows the answer. /model/info reports each alias's backing
model, provider, and declared tool support, so the agent asks at startup and
routes on the provider it discovers. That keeps vendor knowledge where A5
already put it -- in the proxy -- rather than duplicating it as name patterns
here.

There is deliberately no name-based fallback: PASSTHROUGH_PREFIXES and
has_native_adapter are gone, and select_client now requires a catalog. A
second heuristic could only ever disagree with the proxy. An unknown model
raises instead of guessing a route.

Misconfiguration is therefore loud and early. build_agent validates every
role against the catalog before it clones anything, naming all the bad ones
at once. Discovery is per key -- /model/info returns only what the presented
token may use -- so the same check catches a role pointed at a real model
this key cannot reach. An unreachable proxy is reported separately, since an
outage and a typo call for different responses from the operator.

Two things now get recorded rather than inferred: the resolved model is
pinned alongside the role mapping at task start, for the reason A11 already
pins roles, and DelegationStarted carries backing_model so A16 eval history
is not averaged across whatever an alias happened to point at.

A model LiteLLM declares tool-less warns at startup rather than refusing;
A15 expects weak local models to be accommodated, and they are observed
driving tool loops anyway.

Verified live: all four aliases the production key can see route correctly
through discovery, and each chosen route works.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dongar force-pushed alias-discovery from 6df3b6cf90 to e38adf40ca

2026-09-23 01:47:59 +00:00

Compare

dongar merged commit b0d7adb70d into main

2026-09-23 01:49:36 +00:00

dongar deleted branch alias-discovery

2026-09-23 01:49:36 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#35