Pass the capability matrix's models as a pytest option
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 51s

NISSE_CAPABILITY_MODELS was neither a secret nor configuration: which
models to measure is a question for one run, so it is now an argument,
--capability-models=a,b, declared in tests/conftest.py where pytest
reads options at startup. With no models the matrix skips as before, so
CI is unchanged. The last non-secret NISSE_* variable other than
NISSE_CONFIG and NISSE_ENV is gone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-23 23:32:44 +00:00
1 parent 15d69799bf
commit 58b439088f
5 files changed
+53 -25

No files matched your search

+2 -2
View File
@@ -248,8 +248,8 @@ change one thing at a time.
1. **Model and quantization.** Start with whether a candidate holds the
tool-call contract at all:
NISSE_CAPABILITY_MODELS=alias-a,alias-b \
uv run pytest -m integration tests/integration/test_model_capability.py
uv run pytest -m integration tests/integration/test_model_capability.py \
--capability-models=alias-a,alias-b
A model that falls back to prose instead of calling tools is disqualified,
however good its benchmarks look. Before comparing aliases, check what