[v0.4.0] Reproducible instrumentation stack #40
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The bench must be reproducible: fixed seeds, committed configs, committed bench-results.md, per-change notes linking each brain edit to the bench run that validated it, and a way to re-run the exact same comparison later (brain-versus-previous-version harness).
Part of #28 (the midterm).
Partially done: fixed seeds, committed configs, bench-results.md committed per run, per-change notes in issue comments. The gap is the brain-versus-previous-version re-run harness (from #12). Subagent assigned to build it, PR-based.
Subagent started: building the brain-versus-previous-version re-run harness (the deferred piece from #12). PR only.
Progress on #40 (brain-vs-previous-version harness): branch 40-version-harness is cut off master
da67404and clean. I've studied the existing bench machinery (benchPresets has the 13 punishing presets; runBench seeds each game with g*7919 + len(preset)) and confirmed the v0.3.0 brains extract cleanly via 'git show v0.3.0:ai/NAME.lisp'. The harness is now drafted on the host: new benchvs.go that stages the old brains under /tmp/bw-versions/REF/ and maps them into the roster as NAME-old, plus a --bench-vs flag in main.go (bare --bench-vs defaults to v0.3.0). Current step: pushing the two files into the bankdev worktree, making 'go build' and 'go test' pass, then one verification run, commit, push, and the PR.Progress: benchvs.go (extracts old brains via git show, stages to /tmp/bw-versions/REF/, roster NAME-old rotation, per-brain wins, markdown append) and the --bench-vs flag in main.go are wired into the worktree. 'go build' and 'go test' both pass in bankdev. Now running one verification --bench-vs v0.3.0 (40 games x 13 presets) in the background; next is commit + push of branch 40-version-harness, then the PR and reviewer request.
Progress: committed
b23062con branch 40-version-harness (benchvs.go + main.go, 246 insertions) and pushed. gofmt/go vet clean, go build + go test green. One verification --bench-vs v0.3.0 completed in ~30s (520 games): current mains 325 wins vs v0.3.0 194. Opening the PR and requesting review next.Done — PR #43 is up: [#40] Brain-vs-previous-version bench harness (--bench-vs), branch 40-version-harness into master, review requested from robertismo (mergeable, 2 files, +246).
What shipped:
How to run: go build -o bankwars . && ./bankwars --bench-vs v0.3.0 (~30s for a full run). go build + go test are green.
First verification run (v0.3.0): current mains 325 wins vs 194 for v0.3.0 across 1500 plays — mill 198-111, vault 62-21, empire 20-17, adapt 45-45.
Merged: PR #43 (--bench-vs version harness) is on master. First real comparison recorded: current mains 325 wins vs v0.3.0 brains 194 wins across 1500 plays over the 13 punishing presets. The harness exists now, so every future brain change can be re-run against any git ref with fixed seeds. Closing.