I rebuilt my blog's shogi AI in a day with a fleet of AI agents — from bug hunts to WASM and NNUE distillation

The homemade shogi AI on my personal site (meetyudai.com) was, in the words of its owner (an amateur 2-dan player), "way too weak." This is the record of running up to five Claude Code subagents in parallel to see how far I could push it in a single day. It keeps not just what worked but the failed ideas that got mercilessly rejected by A/B testing — the raw diagnostic logs, the catalog of failures, the real numbers, and the lesson (learned the hard way, more than once) that polishing a proxy metric doesn't win you games. Cost: essentially nothing (electricity and LLM usage). This is the overview; the raw logs and code are in the full write-up on my blog.

The gist (three lines up front)

• The real cause of the "nonsense moves" was neither the search nor the eval: an opening-book fallback was bypassing the search engine and answering instantly, a wiring bug. A thinking-time log was the smoking gun.
• Small tweaks to the handwritten engine were "no effect" 8–9 times out of 10 under production time controls. What worked was structural: a TypeScript→WebAssembly port (15× faster search) and, on top of it, NNUE distillation.
• The arc was 19.6% → 32.1% → 77.1% win rate vs the handwritten eval. At the end I lost to it in production, discovered a disease called sigmoid saturation, and cured it by rebuilding the data. Finally the author confirmed, in real games, that it "definitely got stronger."

What I built

I built a shogi AI that runs entirely inside the browser (WebAssembly), with no backend at all. Open the page and it fetches the eval weights (a ~1.1 MB static file) asynchronously; from there it reads a few moves ahead and plays using only your browser's CPU. It runs on a phone browser.

It has two pillars.

The first is the search engine — the "read ahead" part. Shogi averages 80–100+ legal moves per position, and multiplying that across several plies gives astronomical trees. You can't read everything, so you prune obviously-bad branches early. The toolkit: negamax with alpha-beta pruning (cut branches that can't beat what you've already found, assuming the opponent replies optimally), principal-variation search, a transposition table (a cache of already-searched positions), late-move reductions (search later-ordered moves shallower), null-move and futility pruning, a quiescence search (read on until captures settle), and a dedicated mate solver. In short, a box of tools for "cutting corners intelligently so you can read deeper."

The second is the evaluation function — the "eyes" that turn the leaf position of a read into a single number: who's ahead, by how much. No matter how deep the search, it's worthless if it can't score the leaves correctly. It started as a handwritten, rule-based eval; in the end I replaced it with an NNUE (neural-net eval). That's the centerpiece of the story.

The "browser-only" constraint matters more than it sounds. Training uses a GPU, but running it (inference) is CPU-only — and it has to be strong under a harsh budget: a few hundred milliseconds on someone else's phone. That constraint drives every later decision: keep the model tiny, quantize it, speed it up with SIMD.

Why I built it

As a 2-dan player, I found the AI on my site honestly "too weak." Two symptoms.

First, it lost the same way every time to the basic amateur "climbing silver" (bōgin) attack — the ▲2六歩→2五歩→3八銀→2七銀→2六銀→1五銀… march that's on page one of every beginner's book. Losing to that every time isn't a matter of counter-strategy; it means the reading itself is broken.

Second, it occasionally played completely nonsensical moves — dropping a pawn uselessly in its own camp, walking its king into danger. Moves no human would ever make.

The direct motivation was that leaving it broken bugged me. But there was a second one: an experiment — how far can a fleet of AI agents rebuild a piece of software in one day? Shogi AI is an ideal subject, for two clean reasons: a ground truth exists (YaneuraOu, an engine far beyond amateur-top strength), and strength is directly measurable. So you can't escape into "it feels better." Measure, and you get a verdict.

How I ran the agents

Before the technical details, the process itself — because it's half the point.

I ran up to five subagents in parallel. Responsibilities split into "speed up search," "train the eval," "curate the opening book," "harden the code (address review findings)," and "write the article." I stayed the integrator: read each agent's result and decide what to try next, so a human bottleneck doesn't stall the independent work.

The problem is that five agents editing the same repo at once will clobber each other. I solved it by giving each agent its own git worktree — multiple working copies of the same repo. Agent A can rewrite the search code while Agent B touches the training script without stepping on each other's files. Cheap isolation, no merge chaos.

And — most important — every "behavior-changing" change had to pass an A/B test. I trusted no "feels stronger." Each change played N games against the previous version at production time controls, and got adopted only if it genuinely won more. Thanks to this, 8–9 out of 10 ideas I tried were rejected. That's not a failure; it's the whole point. Most plausible ideas don't survive measurement, and building the machine to reject them mechanically, up front, was the single biggest win.

The other mandatory gate was bit-exact parity. Later I ported the engine to WebAssembly, and I refused to allow "faster, but the moves changed slightly." The ported version had to return the same move, same score, same node count, same leaf count as the handwritten TypeScript version — verified on thousands of positions and by exhaustive legal-move counts (perft: 25,440 nodes at depth 3 from the initial position, matching known numbers even with drops in hand). Only the version that matched 100% went to production. Different speed, identical moves — that closed the door on "optimization sneaks in a bug."

Phase 1: reproduce "weak," and find the real culprit

Faced with a weak AI, the temptation is to guess "the eval's bad" and start fixing. But that's a guess. I decided to get hard evidence first.

I turned a losing game — the collapse to bōgin — into a reproducible script as a fixed game record. That script feeds positions, one by one, through the exact production AI path, and logs the returned move, score, material, and how long it thought.

That surfaced the decisive clue: the AI was returning each move in 1–23 milliseconds. Real search of several plies takes hundreds of ms to seconds. An instant 1–23 ms reply means it isn't reading at all.

Chasing it down, the truth was: an opening-book fallback was bypassing the search engine entirely. That mechanism is supposed to instantly return a recorded move for positions that are in the book. But a bug made it return a "plausible-looking" move even for positions outside the book — with no search. So even at critical endgame moments, the AI was choosing moves on vibes, reading nothing. That was the source of the nonsense.

This is the biggest finding. The cause of the weakness was neither the eval nor the search algorithm — it was a wiring bug sitting in front of both. However much you polish eval or search, if the search never gets called, it all gets bypassed. Deleting that fallback eradicated the instant-nonsense class of bug.

The lesson is clear. Inferring the cause (bad eval) from the symptom (bad move) usually misses. Hit it with hard evidence. Thinking time — a seemingly irrelevant number — was what cracked it.

Phase 2: rebuild the search (small tricks barely helped; the structural change did)

With the instant-answer bug gone, I wanted to get stronger while actually reading. I started with the search itself.

I tried textbook tricks: another layer of pruning, better move ordering (searching good-looking moves first makes pruning bite), deeper mate search, tuned margin constants. In theory each should "read deeper" or "cut waste."

But under rigorous A/B at production time controls, most of it was "no effect." Improvements that should deepen the read didn't move the win rate at all. This foreshadows what bit me later: a deeper read doesn't convert to wins if the eval can't correctly judge the positions you reach.

What worked wasn't a trick — it was swapping the substrate. I ported the handwritten TypeScript engine to WebAssembly (WASM), a low-level format that runs at near-native speed in the browser and crunches numbers far faster than JavaScript. Moving the same algorithm to WASM alone gave:

• ~15× faster search
• +3–4 ply of depth at the same time budget
• a 10–0 record against the old engine

In shogi, "+3–4 ply deeper" is a rank's worth of strength. I didn't make the algorithm one bit smarter — just changed the runtime. Speed, not smarts, was the bottleneck — a blunt but crucial lesson.

Phase 3: distill the eval (NNUE)

With search fast and deep, it was time to strengthen the leaves — the "eyes," the evaluation function.

Until then it was handwritten: piece values (a pawn ≈ 100, a rook ≈ 1000…) plus terms for king safety, piece activity, and so on, summed by human-written rules. Readable and decent, but capped at the range of terms a human thinks of. The subtle goodness of a shape can't be fully captured in rules.

So I replaced it with machine learning, via distillation. The idea:

1. Have the strong engine YaneuraOu score millions of positions at fixed depth. That's the teacher data. YaneuraOu is strong but heavy — it can't run in a browser. But as a "teacher who grades your answers," it's ideal.
2. Have a small neural net imitate those scores. The net is deliberately tiny: roughly 2,268 board features → 256 → 32 → 1, with clipped-ReLU activation (a simple non-linearity that caps values that grow too large) and about 590k parameters. Tiny, because it has to produce an eval instantly on a browser CPU. This format is NNUE, and — fun fact — it was originally invented for shogi (chess's Stockfish imported it later). NNUE's strength: when the board changes by only one move, it can update the eval incrementally rather than recompute (the running state that enables this is the "accumulator").
3. Quantize the trained weights to 16-bit integers (round fine decimals to integers), compress to ~1.1 MB, and serve them statically to the browser.

The training target was sigmoid(cp / 600) — the engine's centipawn score (cp = hundredths of a pawn) squashed by a sigmoid into a win-probability-like 0–1. Remember that 600 in the denominator. It bares its teeth in the endgame.

Data: three cycles of measurements

NNUE didn't land in one shot. It took three rounds, each its own little story.

Cycle 1 — a rout (19.6%). The first trained NNUE, in equal-time self-play against the handwritten eval, won just 19.6%. A blowout. Yet the inference was perfectly correct — bit-identical across the training framework, TypeScript, and WASM on hundreds of positions, with matching node counts at equal time (the incremental accumulator worked flawlessly). The implementation was right; the model itself lost.

The post-mortem is the most important lesson here. I'd been judging model quality by "how close it is to the teacher's scores." 2–2.5× closer than the handwritten eval — so it should be stronger, right? But that metric is a bad predictor of match strength. Here's why: what alpha-beta needs is not the absolute eval value but the relative ordering of sibling moves. As long as "this move beats that one" is right, search works. The net's average error was larger than the gap between typical candidate moves (<100cp), so it scrambled the ordering. The handwritten eval, even with an off absolute scale, was self-consistent, so a uniform bias was harmless to search. "Close to the teacher but with jumbled ordering" is weaker in search than "far from the teacher but order-preserving." That lesson held to the end.

Cycle 2 — the reversal (77.1%). Guessing the cause was "error too big to preserve ordering = too little data," I greatly increased the teacher data and retrained. The win rate climbed 19.6% → 32.1% → 77.1%, clearly beating the handwritten eval. More data alone flipped a rout into a comfortable win. I also shipped pondering (thinking on the opponent's clock — a "permanent brain"), lifting mean reached depth 9.00 → 9.35. This finally reached production.

Cycle 3 — speed, then a defeat and its cure. Now that it could win, I made it fast. WASM SIMD (one instruction computing several numbers at once) made the eval 6.2× faster, and I added multithreading (a Lazy-SMP-style parallel search; an estimated +58 Elo, though at n=24 it didn't reach significance, so I recorded it honestly as a point estimate). So far so good. And then — at the top difficulty in production, the author (2-dan) beat the NNUE. The thing I'd just strengthened tripped over its own feet in a real game. The diagnosis revealed the culprit.

The main enemy: sigmoid saturation

Recall the training target was sigmoid(cp / 600). That design hid a trap for the endgame.

A sigmoid flattens and pins near 1 for large inputs, near 0 for small ones. In won or lost positions the centipawn score is huge (say +3000cp), so sigmoid(3000/600)=sigmoid(5) is essentially 1.0 — maxed out. The problem: from there, moving the eval a bit — +3000 or +3500 — barely changes the output. So moves that should differ in value all collapse to nearly the same number.

In the problem position from the diagnosis, all 71 legal moves collapsed into a 15cp band. To the AI, "every move is the same." So in a position where the win (or loss) was clear, it picked essentially at random and produced nonsense. The "nonsense move" in the game the author lost came from exactly this. Ironically, making it read deeper made things worse, by reaching more of those saturated positions.

The fix was in the data, not the search. The earlier training data had thinned out decisive (lopsided) positions as "boring" — and that thin region was exactly where saturation lived. So I retrained with decisive positions deliberately over-sampled — up to about half the set — on 5.24M positions (labels at depth 12), to teach it the saturated region and let it tell tiny differences apart inside a big lead. Results:

• Move-value spread in the problem position recovered from 20cp → 532cp (26×): "all the same" became "clearly different."
• Blunders in that position halved, 8 → 4.
• 92.2% win rate vs the old NNUE; 84.4% vs the handwritten eval (at both 1000 ms and 2000 ms).
• And finally, the author confirmed in real games that it "definitely got stronger."

Not a proxy metric — the author playing it and admitting it. That was the only pass/fail I trusted.

What I learned (failures and all)

Honestly, the biggest payoff of the day was the record of failures, more than the wins.

A good proxy metric doesn't mean you win. Doubling "closeness to the teacher" didn't make it stronger. The final judge is always the same: do you win the actual game — ideally against a human playing the openings you actually face? Bench numbers are only a mid-way signal.

Verification itself is full of traps. Self-play statistical degeneration (self-play skews the win/loss when both sides share quirks), time-control bias (the budget you measure at changes the conclusion), and a config mismatch between old and new defaults — each nearly led me to "conclude it got stronger" by mistake. The only defense is to measure under production-equivalent conditions.

Catalog of what didn't work — a memorial to the plausible ideas that died on measurement. King-piece (KP) features (a powerful method adding fine king-to-piece relations, but at 1M positions it made things worse via data dilution), check extensions (lost the A/B), drop-move late-move pruning (same), deepening the mate solver (no change). The capstone: full bitboards. Holding the board as bits of an integer and processing with bit operations is the core of a native engine's speed. I tried to copy it — but in 32-bit JavaScript the 81 squares don't fit in a single integer. Resolving sliding-piece attacks straddles words awkwardly, and a slider-list prototype came out ~2× slower than the array ray-walk the JIT already optimizes well. Theoretically faster, slower here. A genuinely different arena from native C++.

How I'd make it stronger from here

Honestly: the browser arena has a ceiling. The search side is done for practical purposes in this environment. With SIMD, multithreading, and bit-exact speedups all in, there's no structural "depth jump" left to grab.

To go further, the lever is the eval (data) side. The strongest candidate is a self-play data loop — have the current strong engine play itself to gather "positions that arise in genuinely strong play," re-label them with YaneuraOu, and retrain, closing a positive loop. Caveat: the grader is still YaneuraOu, so the ceiling is YaneuraOu's judgment (this is not true self-improvement à la AlphaZero, which learns from game outcomes themselves). The other lever is hunting concrete weaknesses the author finds in real games. The saturation discovery was exactly that: a single real game pointed at the weak spot more sharply than any proxy metric could.

Cost and wrap-up

Total cost: electricity and LLM usage only. Time: effectively one day. From "way too weak" to "the author admits it's definitely stronger," reached by running AI agents in parallel plus the unglamorous discipline of A/B testing and bit-exact verification.

Looking back, this isn't a story of "one clever discovery." Quite the opposite: measure, discard the failures, keep only what survives — that boring repetition is what worked. 8–9 out of 10 flashy ideas died; the surviving few (the WASM port, more data, killing saturation) carried the whole thing. Reliably rejecting the changes that make it weaker mattered far more than being clever.

The full version (raw logs, code, all the numbers)

Everything above is a digest. The raw debugging logs, the details of every failed approach, the per-cycle measurement graphs, and the code are all in the full write-up. There's also a beginner primer explaining NNUE, distillation, backprop, and quantization from scratch.

👉 Full write-up: https://www.meetyudai.com/blog/applied-algorithms/9fpLthtYgHvnImuFR0Jf 👉 Primer (for ML beginners): https://www.meetyudai.com/blog/applied-algorithms/H9beqVVY7h5qc3hNfskR