Bench nondeterminism: identical-binary re-runs differ in bond-economy and micro-economy (+-1 win) #100
Winnie
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found independently by exp11 and exp16 during #97 round 2/3, verified both times from the bw85 container.
Symptom
Two runs of the SAME built binary, same bench settings (30 games x 13 presets, gauntlet 2), differ in exactly two presets: bond-economy and micro-economy. Magnitude is small: +-1 win (e.g. adapt 14<->15, super 6<->7 in micro-economy), avg-equity shifts of ~5-13. The other 11 presets are byte-identical across runs.
What was ruled out
The source is unidentified and somewhere else in the bench harness (or the auction/bankruptcy paths).
Impact
The team quotes wins per preset to +-1 in round tables; bond-economy and micro-economy totals should not be treated as exact until this is bisected. The win totals across the other presets are stable (e.g. exp16's tight-cash 7/30 reproduced across 4 runs).
Refs: #97 (live work), exp11.md and exp16.md in /root/bw85-reports/ on the host.
Update from exp19 and exp23 (round 4) — stronger evidence, upgraded severity
exp19 measured: two binaries (canonical v0 vs v2) produce BYTE-IDENTICAL games in mega-loans and cash-glut (full-block diff of the 30 mega games: 0 differing lines out of 6,120), yet the aggregate rows differ (super mega avg equity 15,511 vs 16,008; +1 win). Both binaries are individually run-stable. Since the games are provably identical but res.wins/res.avgEq differ, the divergence must live in the bench's in-memory aggregation — prime suspect: the winner is decided by a strict > on b.LastReport.Equity while the printed ranking uses the recomputed end-of-game g.report(b).
exp23 found the cross-game cousin: the same canonical file gives micro-economy adapt 15/super 6 when games run standalone (clean process) but adapt 14/super 7 when the preset runs after others in the same bench process. A cross-game stateful leak in NewGame or the report path is the likely second source.
Impact on the team: treat +-1 rows in mega-loans, cash-glut, micro-economy, bond-economy as untrusted until this is bisected. Win deltas of +3 or larger in other presets are still meaningful. No experimenter is allowed to touch bench.go (team rule), so this needs an engine-side bisect.
Fixed. PR #101 (branch 100-nondet @
73fdb47), review requested.Root cause: the loan shelf (
openLoanOffers) is the one phase-scratch global read before it's written — turn-1 deposit hooks see(market-loans)before the game's shelf opens, and the bench's map-random preset order let the PREVIOUS preset's last shelf bleed into the next game's first turn. That flipped close games in the attrition-heavy presets.Fix:
run()clears the shelf at game start. Verification: two identical 30-game benches that differed by 84 lines before the fix (the adapt 14<->15 and super 6<->7 flips) are now byte-identical (0 diff lines). Regression test plants a stale shelf and asserts turn 1 sees an empty market. Full suite + vet clean.Root cause found (exp23) — cross-game shelf leak
exp23 traced the leak:
openLoanOffers(the Go global behind(market-loans)) is never cleared at game start, so turn 1 of game N+1 sees game N's leftover shelf. In the polluted micro-economy game, super's env held pool-est=1200 / prev-deposits=20000 — deposit-drought's config values, the preset that ran earlier in the same bench process. Same mechanism explains the earlier ±1 rows and the aggregation-row divergences exp19 measured (winner picked from b.LastReport while the printed ranking recomputes g.report).Impact: every number measured on this engine carries turn-1 pollution, including the trunk 100g 389/1300. exp23's suggested fix: clear
openLoanOffersin NewGame/assignPlayers, or have(market-loans)return empty outside the loan phase. Then re-baseline and re-measure everything.Dispatching an engine-fix agent to implement + regression-test this on its own branch with a PR for review.
Engine fix shipped as PR #102 (branch 100-shelf-reset, #102), review requested from robertismo.
What shipped:
NewGamenow nilsopenLoanOffersat the construction boundary. PR #101 cleared it atrun()start, which fixed turn-1 hook reads, but brain files load inassignPlayersbeforerun(), and defvar initializers evaluate AT LOAD, so a load-time read like(defvar pool-est (length (market-loans)))still caught the previous game's shelf and carried it for the whole game. The construction-boundary clear closes that path too;run()'s clear remains as a second guard.Deliberately did NOT gate
(market-loans)outside the loan phase: within a running game the shelf is the last-turn loan book, the same information regime as the #84 deposit/bond snapshots. Gating it would change what brains observe mid-game and shift bench numbers. The leak is a cross-game problem and the game-boundary clear fully closes it.Regression test:
TestNewGameClearsShelfBeforeBrainLoadmirrors the bench path — plants a stale shelf, constructs a new game, loads a probe brain whose defvar captures the shelf length at load, posts it on turn 1, and asserts 0. Verified red on master (posted 0.5000, the capped nonzero stale read) and green with the fix; it also asserts the hook really fired and the shelf rebuilt, so it can't pass vacuously.Gates: go build OK, go test -count=1 . OK (includes #101's shelf test), go vet ./... OK. Diff is game.go + shelf_load_test.go only.