End-to-end tests on a local model, and in CI on every PR
- Every e2e role runs on qwen3.8-27b on the RTX 4090 (e2e_roles(); NISSE_E2E_MODEL overrides, e.g. claude-haiku-4-5). The first full run passed 7 of 8 in 26 minutes for $0.0000; the eighth was a real finding, fixed with the new-project PR. - scripts/e2e.sh reports a run's spend on the test key and the day's total against its budget: above $0 on the local model means a paid fallback. - CI runs the suite as its own job on pull requests (not pushes to main), on a throwaway sshd (--local), cancelling a run a newer push supersedes, so the checks job still reports in minutes. - The plan says so: e2e on every PR; locally only -k / --lf. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
b3bf6d5a6e
commit
6db80dfb4b
6 files changed
+91
-9
No files matched your search
@@ -54,3 +54,44 @@ jobs:
|
||||
# key) and deletes it afterwards.
|
||||
NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
|
||||
run: uv run pytest -m "integration and not e2e" -rs
|
||||
|
||||
e2e:
|
||||
# The end-to-end suite (scripts/e2e.sh): the whole agent against real
|
||||
# services, every role on a local model, so it spends nothing but keeps
|
||||
# the RTX 4090 busy for ~25 minutes. On PRs only, as its own job so the
|
||||
# checks above report first; a newer push to the same PR cancels the
|
||||
# run it supersedes. Locally, run only what you're writing (-k) or
|
||||
# diagnosing (--lf).
|
||||
if: github.event_name != 'push'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
concurrency:
|
||||
group: e2e-${{ github.head_ref || github.ref }}
|
||||
cancel-in-progress: true
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install uv
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
|
||||
|
||||
- name: Sync workspace
|
||||
run: uv sync --all-packages
|
||||
|
||||
- name: Install sshd
|
||||
# The build VM is a throwaway sshd here (scripts/e2e.sh --local).
|
||||
run: |
|
||||
SUDO=$(command -v sudo || true)
|
||||
$SUDO apt-get update -qq
|
||||
$SUDO apt-get install -y -qq openssh-server openssh-client > /dev/null
|
||||
$SUDO mkdir -p /run/sshd
|
||||
|
||||
- name: End-to-end tests
|
||||
env:
|
||||
NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }}
|
||||
NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }}
|
||||
NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }}
|
||||
NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
|
||||
NISSE_E2E_RUNS: /tmp/nisse-e2e-runs
|
||||
run: scripts/e2e.sh --local -v
|
||||
Reference in new issue
Block a user