Move configuration to YAML, and make the tuning knobs configurable #43

2026-09-23T23:14:07Z

dev-agent commented

2026-09-23 23:14:07 +00:00

Closes #32. This is the first of the 4090-readiness changes.

Config file

NISSE_CONFIG, defaulting to ./nisse-dev.yaml. See nisse-dev.example.yaml, which documents every key.
It's the only home for settings. The environment supplies only the secrets the file references by ${…}, plus NISSE_CONFIG and NISSE_ENV to find the files. The old per-setting variables and the from_env() constructors are gone.
A reference to an unset variable is an error, never an empty string.
Fails fast, reporting everything at once. Unknown keys are errors, so a misspelt knob fails at startup instead of silently doing nothing.
The config file is added to the check that refuses to start when files sit inside the workspace clone.
Spec A5 now says config comes from the file.

Tuning knobs

These were hardcoded; the defaults are the values the agent already used:

executor.max_turns, executor.failure_limit
expert.consult_budget
delegation.max_turns, delegation.max_tokens, delegation.wall_clock_s
tools.command_timeout_s, tools.max_output_chars
llm.max_output_tokens

Integration tests and CI

tests/integration/nisse-test.yaml is a committed agent config aimed at test resources: the testing identities, the test room and the sandbox repo. Its tokens are ${…} references.
One conftest.py loads that config and builds the clients.
NISSE_LIVE_PLANNER_MODEL becomes the file's planner role. Every role is claude-haiku-4-5, including the executor for the LiteLLM smoke test.
NISSE_CAPABILITY_MODELS becomes a pytest option, --capability-models=a,b, because which models to measure is decided per run. With no models given, the matrix skips as before.
A missing secret now fails the live tests instead of skipping them.
CI passes only NISSE_LLM_TOKEN, NISSE_MATRIX_TOKEN and NISSE_GITEA_TOKEN. After merging, the other eight Actions secrets can be deleted.

`docs/Tuning-Guide.md`

Written for learning local-model tuning:

How weights, quantization and the KV cache use VRAM, with the per-token formula and worked numbers for the candidate models.
Prefill vs decode, and prefix caching.
Each knob: what goes wrong at either extreme, and what evidence to look at.
An order to tune in, and a warning about run-to-run noise.

It also notes a weakness: output truncation keeps the start, but pytest prints its failure summary at the end. That's a candidate for the context-budget PR.

Tests

Config: tree and file loading, ${…} resolution including a missing secret, every error reported at once, unknown keys, non-numbers and zeros.
Startup: a missing file refused with a pointer to the example; files refused when they're inside the clone; the example file loads.
Tuning: every value reaches its component.
scripts/check.sh passes locally: 343 unit tests, and the integration suite with 17 passed and 5 skipped, all against the committed test config.

New dependency: pyyaml.

🤖 Generated with Claude Code

Closes #32. This is the first of the 4090-readiness changes. ## Config file - `NISSE_CONFIG`, defaulting to `./nisse-dev.yaml`. See `nisse-dev.example.yaml`, which documents every key. - **It's the only home for settings.** The environment supplies only the secrets the file references by `${…}`, plus `NISSE_CONFIG` and `NISSE_ENV` to find the files. The old per-setting variables and the `from_env()` constructors are gone. - **A reference to an unset variable is an error**, never an empty string. - **Fails fast, reporting everything at once.** Unknown keys are errors, so a misspelt knob fails at startup instead of silently doing nothing. - The config file is added to the check that refuses to start when files sit inside the workspace clone. - Spec A5 now says config comes from the file. ## Tuning knobs These were hardcoded; the defaults are the values the agent already used: - `executor.max_turns`, `executor.failure_limit` - `expert.consult_budget` - `delegation.max_turns`, `delegation.max_tokens`, `delegation.wall_clock_s` - `tools.command_timeout_s`, `tools.max_output_chars` - `llm.max_output_tokens` ## Integration tests and CI - `tests/integration/nisse-test.yaml` is a committed agent config aimed at test resources: the testing identities, the test room and the sandbox repo. Its tokens are `${…}` references. - One `conftest.py` loads that config and builds the clients. - `NISSE_LIVE_PLANNER_MODEL` becomes the file's `planner` role. Every role is `claude-haiku-4-5`, including the executor for the LiteLLM smoke test. - `NISSE_CAPABILITY_MODELS` becomes a pytest option, `--capability-models=a,b`, because which models to measure is decided per run. With no models given, the matrix skips as before. - **A missing secret now fails the live tests** instead of skipping them. - CI passes only `NISSE_LLM_TOKEN`, `NISSE_MATRIX_TOKEN` and `NISSE_GITEA_TOKEN`. **After merging, the other eight Actions secrets can be deleted.** ## `docs/Tuning-Guide.md` Written for learning local-model tuning: 1. How weights, quantization and the KV cache use VRAM, with the per-token formula and worked numbers for the candidate models. 2. Prefill vs decode, and prefix caching. 3. Each knob: what goes wrong at either extreme, and what evidence to look at. 4. An order to tune in, and a warning about run-to-run noise. It also notes a weakness: output truncation keeps the *start*, but pytest prints its failure summary at the *end*. That's a candidate for the context-budget PR. ## Tests - Config: tree and file loading, `${…}` resolution including a missing secret, every error reported at once, unknown keys, non-numbers and zeros. - Startup: a missing file refused with a pointer to the example; files refused when they're inside the clone; the example file loads. - Tuning: every value reaches its component. - `scripts/check.sh` passes locally: 343 unit tests, and the integration suite with 17 passed and 5 skipped, all against the committed test config. New dependency: `pyyaml`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-23 23:14:08 +00:00

Move configuration to a YAML file, and make the tuning knobs configurable

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 51s

Details

df449c7e34

The thresholds Part H says to tune on the real inference hardware were
hardcoded, so every adjustment would have been a PR. They now live in a
`tuning:` section of a YAML config file, which also replaces flat
environment variables as the home for structure (#32).

- The file (NISSE_CONFIG, default ./nisse-dev.yaml) carries structure
  and holds no secrets: credentials are ${VAR} references resolved from
  the environment, and a reference to an unset variable is an error
- The environment still wins: every existing NISSE_* variable overrides
  its key, and the file is optional, so .env and CI work unchanged
- Validation stays fail-fast and reports every problem at once; unknown
  keys are errors, so a misspelt knob fails at startup instead of
  silently doing nothing
- The config file joins the state store and .env in the check that
  refuses to start when they sit inside the workspace clone
- Tuning covers the executor's turn and failure limits, the consult
  budget, the delegation caps, the command timeout and the tool output
  limit, which Workspace now takes as a setting; the defaults are the
  values the agent already ran with
- llm.max_output_tokens reaches both LLM clients from the config

docs/Tuning-Guide.md explains the knobs for someone learning local
model tuning: how weights, quantization and the KV cache spend VRAM,
what each knob does and what goes wrong at either extreme, and an order
to tune in. nisse-dev.example.yaml documents every key and is checked
by a test so it cannot drift.

Closes #32

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-23 23:25:59 +00:00

Make the config file the only home for settings

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 51s

Details

15d69799bf

With no production instance yet, there is no reason to carry two ways
of configuring the agent. Every environment variable that duplicated a
config key is gone; the environment now supplies only the secrets the
file references by ${…}, plus NISSE_CONFIG and NISSE_ENV to find the
files.

- AgentConfig loads from a file (load, from_file) or a parsed tree
  (from_tree, for tests); a missing file is an error naming the example
- The from_env constructors on LiteLLMClient, AnthropicPassthroughClient,
  GiteaClient, MatrixConfig and RoleMap are removed, with the env
  reading for max output tokens; clients take their settings from config
- The live integration tests share a committed agent config,
  tests/integration/nisse-test.yaml, aimed at test resources, and one
  conftest that loads it and builds clients. NISSE_LIVE_PLANNER_MODEL
  becomes its planner role; NISSE_CAPABILITY_MODELS stays as a per-run
  switch
- A missing secret now fails the live tests instead of skipping them
- CI passes only the three tokens
- .env.example is secrets only; spec A5 says config comes from the file

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-23 23:32:46 +00:00

Pass the capability matrix's models as a pytest option

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 51s

Details

58b439088f

NISSE_CAPABILITY_MODELS was neither a secret nor configuration: which
models to measure is a question for one run, so it is now an argument,
--capability-models=a,b, declared in tests/conftest.py where pytest
reads options at startup. With no models the matrix skips as before, so
CI is unchanged. The last non-secret NISSE_* variable other than
NISSE_CONFIG and NISSE_ENV is gone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar reviewed

2026-09-23 23:36:59 +00:00

.env.example

						
				@@ -3,0 +3,4 @@

				#

				# Everything else lives in the YAML config (see nisse-dev.example.yaml),

				# which refers to these by name, e.g. `token: ${NISSE_LLM_TOKEN}`. The live

				# integration tests use the same names, from tests/integration/nisse-test.yaml.

dongar commented

2026-09-23 23:36:59 +00:00

I feel like this should include some examples of the actual secrets we expect that match the example config file. Something like:

NISSE_MATRIX_TOKEN=replace-me
NISSE_GITEA_TOKEN=replace-me
NISSE_LLM_TOKEN=replace-me

I feel like this should include some examples of the actual secrets we expect that match the example config file. Something like: NISSE_MATRIX_TOKEN=replace-me NISSE_GITEA_TOKEN=replace-me NISSE_LLM_TOKEN=replace-me

dongar approved these changes

2026-09-23 23:42:50 +00:00

Dismissed

dongar requested changes

2026-09-23 23:44:09 +00:00

Dismissed

dongar left a comment

Please address my comments before submitting.

dev-agent reviewed

2026-09-23 23:45:02 +00:00

.env.example

						
				@@ -59,1 +4,3 @@

				# NISSE_ENV=~/nisse-demo-workspace/.env

				# Everything else lives in the YAML config (see nisse-dev.example.yaml),

				# which refers to these by name, e.g. `token: ${NISSE_LLM_TOKEN}`. The live

				# integration tests use the same names, from tests/integration/nisse-test.yaml.

dev-agent commented

2026-09-23 23:45:02 +00:00

They're in the file, just below this comment block. The three lines didn't change in this PR, so the diff folded them away. To make the match with the example config explicit, each one now notes which key refers to it (llm.token, matrix.token, gitea.token), and the values are replace-me as you suggested. Done in e1034a2c.

They're in the file, just below this comment block. The three lines didn't change in this PR, so the diff folded them away. To make the match with the example config explicit, each one now notes which key refers to it (`llm.token`, `matrix.token`, `gitea.token`), and the values are `replace-me` as you suggested. Done in e1034a2c.

dev-agent added 1 commit 2026-09-23 23:45:04 +00:00

Show which config key refers to each example secret

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 49s

Details

e1034a2cb7

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar approved these changes

2026-09-23 23:49:41 +00:00

dongar merged commit 31851b7396 into main

2026-09-23 23:49:44 +00:00

dongar deleted branch yaml-config

2026-09-23 23:49:45 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#43