Spec: park vision as a need of future agents, not current ones
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m20s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
50a2a75342
commit
9239a56bd1
1 file changed
+12
-21
@@ -1424,9 +1424,10 @@ The rest resolve when reached, not now.
|
|||||||
and know nothing about which model is loaded. (Measured 2026-09-28: they only
|
and know nothing about which model is loaded. (Measured 2026-09-28: they only
|
||||||
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
|
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
|
||||||
- **Preferred: one local model for every local role.** No swaps at all. It
|
- **Preferred: one local model for every local role.** No swaps at all. It
|
||||||
needs one model good enough as the dev executor, for ops' Home Assistant tool
|
needs one model good enough as the dev executor and for ops' Home Assistant
|
||||||
calls, and for vision (next item). The executor benchmark and the vision and
|
tool calls. The executor benchmark and the ops evaluation decide whether one
|
||||||
ops evaluations decide whether one exists.
|
exists. More agents are planned for other jobs, which makes one shared model
|
||||||
|
more valuable still, and some of them may add vision (next item).
|
||||||
- **Fallback: agents tag in and out.** If no single model fits, the agents
|
- **Fallback: agents tag in and out.** If no single model fits, the agents
|
||||||
coordinate their use of the host rather than interleave: an agent holds the
|
coordinate their use of the host rather than interleave: an agent holds the
|
||||||
host for a stretch of work on its model, then hands it over, so a swap happens
|
host for a stretch of work on its model, then hands it over, so a swap happens
|
||||||
@@ -1434,24 +1435,14 @@ The rest resolve when reached, not now.
|
|||||||
groups), where VRAM allows, can reduce how often that's needed. How the
|
groups), where VRAM allows, can reduce how often that's needed. How the
|
||||||
hand-over works (a lease, a queue, or priority for ops alerts) is only worth
|
hand-over works (a lease, a queue, or priority for ops alerts) is only worth
|
||||||
designing if the preferred option fails.
|
designing if the preferred option fails.
|
||||||
- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the
|
- **Parked: vision, for future agents.** More agents are planned for other kinds
|
||||||
model must look at a rendered part to check it against the design, and for
|
of jobs, and some will need vision: checking OpenSCAD renders against a design,
|
||||||
video-file management planned later (which agent owns it is not yet
|
or managing video files. Neither current agent needs it. The local candidates on
|
||||||
decided). The local candidates on the
|
the 4090 host already accept images, and video as sampled frames, but the agent
|
||||||
4090 host accept images, and video as sampled frames, but the agent is
|
code is text-only. Building it will need tools that return images, image content
|
||||||
text-only: canonical events carry strings, and neither renderer can send an
|
in canonical events (A10) and both renderers, storage that keeps the durable log
|
||||||
image. Needed before any vision work:
|
(A2) small, and image tokens in the context budget. How well a model sees is
|
||||||
- a way for a tool to return an image, e.g. an OpenSCAD render or a frame from
|
measured when that work starts, not before.
|
||||||
a video file;
|
|
||||||
- image content in the canonical event model, rendered as OpenAI image parts
|
|
||||||
and Anthropic image blocks, so delegation to a frontier model keeps it;
|
|
||||||
- storage that keeps the durable log (A2) small, e.g. a reference and hash
|
|
||||||
rather than the image bytes;
|
|
||||||
- image tokens counted in the context budget; video frames multiply them.
|
|
||||||
Which local model is good enough at seeing is measured separately from
|
|
||||||
tool calling: an evaluation with rendered OpenSCAD parts carrying seeded
|
|
||||||
mistakes, and sample video frames.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Repository Layout
|
## Repository Layout
|
||||||
|
|||||||
Reference in new issue
Block a user