Catan LLM arena

every match on disk, replayed from its move log · analysis ↓

what this is

Four language models played 50 games of Catan against each other. No bots, no humans, no judges — every move chosen by the model whose seat it was, retried until it answered, never replaced by a random one.

50games
23,911moves
9,484model decisions
$5.54total spend

DeepSeek and Luna took 38 of 50 games (p = 0.0003). But no single model beats chance once you correct for testing four of them, and 19–19 does not separate the leaders.

The catch: they never saw the board — only a numbered list of legal moves, and the engine happens to list the best spots first. Read the ranking with that in mind.

full statistics · source, and the review that found three of my mistakes

models

modelwinsgamesmovesretriesmedian decisionspend
gpt-5.6-luna1950253525s$1.225
deepseek-v4-flash19502416363549s$1.271
gemini-3.7-flash750236633s$1.425
hy3550216759114s$1.622
Timings are whole-decision wall clock — they include failed attempts and backoff, not model latency alone. A model that retries often reads slower here than it answers.

matches

gameturnmovesseatsstatus
m000_g0234676deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: gemini-3.7-flash
m001_g0194586hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: gemini-3.7-flash
m002_g091278gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: gpt-5.6-luna
m003_g0127402gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: gpt-5.6-luna
m004_g0189598deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: hy3
m005_g0134439hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: hy3
m006_g0218664gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: deepseek-v4-flash
m007_g0122373gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: hy3
m008_g0154470deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: gpt-5.6-luna
m009_g0212664hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: gemini-3.7-flash
m010_g0105322gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: deepseek-v4-flash
m011_g073214gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: gpt-5.6-luna
m012_g0165531deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: gpt-5.6-luna
m013_g0100288hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: gpt-5.6-luna
m014_g0156480gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: gemini-3.7-flash
m015_g0132426gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: gpt-5.6-luna
m016_g0157484deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: gpt-5.6-luna
m017_g0110330hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: deepseek-v4-flash
m018_g0131425gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: gemini-3.7-flash
m019_g0150445gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: gpt-5.6-luna
m020_g0248742deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: deepseek-v4-flash
m021_g0183557hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: deepseek-v4-flash
m022_g0128423gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: gpt-5.6-luna
m023_g0146465gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: deepseek-v4-flash
m024_g0194581deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: gemini-3.7-flash
m025_g0214623hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: gpt-5.6-luna
m026_g0106330gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: gpt-5.6-luna
m027_g070225gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: gpt-5.6-luna
m028_g0147451deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: gpt-5.6-luna
m029_g03551018hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: deepseek-v4-flash
m030_g0143414gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: hy3
m031_g0137439gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: gpt-5.6-luna
m032_g0198614deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: deepseek-v4-flash
m033_g0207630hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: deepseek-v4-flash
m034_g058170gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: gemini-3.7-flash
m035_g0204616gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: gpt-5.6-luna
m036_g0114346deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: hy3
m037_g0139415hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: deepseek-v4-flash
m038_g0241693gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: deepseek-v4-flash
m039_g0160475gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: deepseek-v4-flash
m040_g0180580deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: deepseek-v4-flash
m041_g0129422hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: gpt-5.6-luna
m042_g0102347gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: deepseek-v4-flash
m043_g0100325gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: gpt-5.6-luna
m044_g0151448deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: deepseek-v4-flash
m045_g0214588hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: deepseek-v4-flash
m046_g0187574gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flash, hy3winner: deepseek-v4-flash
m047_g0113370gpt-5.6-luna, deepseek-v4-flash, hy3, gemini-3.7-flashwinner: gpt-5.6-luna
m048_g0184609deepseek-v4-flash, hy3, gemini-3.7-flash, gpt-5.6-lunawinner: deepseek-v4-flash
m049_g0109326hy3, gemini-3.7-flash, gpt-5.6-luna, deepseek-v4-flashwinner: deepseek-v4-flash