[#46] Structured iteration log: every bench run machine-readable #48
No reviewers
Labels
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Winnie/bankwars!48
Loading…
Reference in a new issue
No description provided.
Delete branch "46-iter-log"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What changed
Adds a structured, machine-readable iteration log to the bench harness (issue #46). Every
--benchand--bench-vsrun now appends one record toiterations.jsonin the repo root, committed alongsidebench-results.md. The human-readablebench-results.mdoutput is unchanged.iterlog.go: package-levelIterationRecord,PresetRow,GauntletRow,VSRowtypes;appendIterationloads the existing file, appends, and rewrites — old records are never lost, and a corrupt log is reported as an error instead of being silently clobbered.bench.go:runBenchcaptures per-preset per-brain rows (wins/plays/avg equity/maxAbs) viapresetRowFromas each preset finishes, andrunGauntletreturns per-specialist rows (wins/plays/presets-won) as it computes the existing per-preset print lines.benchvs.go:runBenchVsappends a record whoseBenchVSsection carries current-vs-old wins/plays per brain.main.go:--benchgains an optional hypothesis argument (--bench "text"), plus a--hypothesis text/--hypothesis=textflag; absent →"untagged run".git rev-parse HEAD(exec.Command).BANKWARS_BENCH_GAMES/BANKWARS_GAUNTLET_GAMESshrink game counts for tiny smoke benches of the real binary (defaults unchanged: 100 / 40).presetRowFromconversion, and an end-to-end 1-game bench that asserts a complete record lands initerations.json.iterations.json schema
Top level: a JSON array of iteration records, one per bench run.
presets: keyed by preset name; per-brain wins, plays, average final equity, max |equity|.gauntlet: keyed by specialist name; total specialist wins/plays and number of presets the specialist won at least once in.bench_vs: keyed by brain name; current vs old wins/plays (empty object on plain bench runs).Verification
No full bench run. Verified with tiny real-binary invocations (
BANKWARS_BENCH_GAMES=1 ./bankwars --bench "text"etc.): inline-hypothesis, separate-flag, untagged default, bench-vs records, and 4 consecutive runs preserving all 4 records.go build/go vet ./.../go test -count=1 .green (10 tests).@Winnie can you show me an example of the output of this?
Here is a real record, produced by running the actual binary on the branch (2 games per preset, 1 gauntlet game per specialist, hypothesis passed as --bench "example run for Robert: schema demo"):
{
"hypothesis": "example run for Robert: schema demo",
"git_sha": "
4bf460e","timestamp": "2026-09-29T18:35:13Z",
"presets": {
"base": {
"wins": {"mill": 2, "adapt": 0, "empire": 0, "vault": 0},
"plays": {"mill": 2, "adapt": 2, "empire": 2, "vault": 2},
"avg_eq": {"mill": 29677.9, "vault": 4674.7, "adapt": 3186.7, "empire": 496.8},
"max_abs": {"mill": 57740.8, "vault": 8800.0, "adapt": 6124.9, "empire": 983.9}
},
"bond-economy": { ... wins: vault 2 ... },
... (all 13 presets)
},
"gauntlet": {
"loan-bond-spec": {"wins": 6, "plays": 26, "presets_won": 6},
"loan-spec": {"wins": 3, "plays": 26, "presets_won": 3},
"auction-spec": {"wins": 1, "plays": 26, "presets_won": 1},
"deposit-auction-spec": {"wins": 1, "plays": 26, "presets_won": 1},
"deposit-loan-spec": {"wins": 1, "plays": 26, "presets_won": 1},
"loan-auction-spec": {"wins": 1, "plays": 26, "presets_won": 1},
"bond-auction-spec": {"wins": 0, ...}, "bond-spec": {"wins": 0, ...},
"deposit-bond-spec": {"wins": 0, ...}, "deposit-spec": {"wins": 0, ...}
},
"bench_vs": {}
}
Every future --bench or --bench-vs run appends one record like this to iterations.json in the repo (old records are loaded and preserved; a corrupt file errors instead of clobbering). The #47 PDF pipeline reads exactly this to build the report tables, so nothing in the report is hand-typed. Full production runs use 100 bench games / 40 gauntlet games per preset.
@Winnie why did you not give me a raw output, I want to literally see it, paste it directly in a json block.
Here is the raw record, verbatim from the binary on the branch (2 bench games / 1 gauntlet game per preset):