Pass the capability matrix's models as a pytest option
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 51s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 51s
NISSE_CAPABILITY_MODELS was neither a secret nor configuration: which models to measure is a question for one run, so it is now an argument, --capability-models=a,b, declared in tests/conftest.py where pytest reads options at startup. With no models the matrix skips as before, so CI is unchanged. The last non-secret NISSE_* variable other than NISSE_CONFIG and NISSE_ENV is gone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
15d69799bf
commit
58b439088f
5 files changed
+53
-25
No files matched your search
@@ -248,8 +248,8 @@ change one thing at a time.
|
||||
1. **Model and quantization.** Start with whether a candidate holds the
|
||||
tool-call contract at all:
|
||||
|
||||
NISSE_CAPABILITY_MODELS=alias-a,alias-b \
|
||||
uv run pytest -m integration tests/integration/test_model_capability.py
|
||||
uv run pytest -m integration tests/integration/test_model_capability.py \
|
||||
--capability-models=alias-a,alias-b
|
||||
|
||||
A model that falls back to prose instead of calling tools is disqualified,
|
||||
however good its benchmarks look. Before comparing aliases, check what
|
||||
|
||||
Reference in new issue
Block a user