I tried to replace an LLM with a 17 MB CNN for reading handwritten chess scoresheets

Francisco López · 1 September 2026 · about 8 minutes

At a chess tournament you write your moves on paper. Forty, sixty, sometimes a hundred half-moves, in pencil, in a hurry, with crossings-out. Then you go home and type them into a computer, one by one, if you want to keep the game.

I built an app that photographs the sheet and gives you the game back. It works: an LLM reads the handwriting and a solver checks every candidate move against the position. What it also does is cost money — about €0.13 per scoresheet. So I spent two weeks trying to replace it with a small model I could eventually run on the phone.

It didn't work. Here is the whole thing, including the parts where I was measuring the wrong number.

The first mistake: chess notation is not handwriting

I reached for TrOCR (microsoft/trocr-base-handwritten) without thinking about it. Handwriting recognition, handwritten input, done.

TrOCR is trained on IAM — handwritten letters, prose — and it decodes into sub-word tokens with a language model inside. Faced with a chess move it does what a language model does: it looks for words.

Bf4 → "afy"     Be2 → "aez"     Nc6 → "cco"
b3  → "63"      Nb4 → "cbt"     d4  → "104"

It transliterates digits into letters, systematically. Shown bxc5, it once produced types.

The correction came from the user of my own app, who happens to be me at a chessboard: SAN moves are not writing, they are symbols that happen to use letters and digits. Nobody writes Nf3 as part of a sentence. The right family is scene-text recognition — licence plates, serial numbers — which decodes character by character and has no opinion about what words exist.

Switching to a CRNN with CTC decoding (docTR's crnn_vgg16_bn), on the exact same photos, without training anything:

140× faster, and better. A whole scoresheet in seconds instead of thirteen minutes, which is what makes an on-device version imaginable at all.

There is a second, deeper reason CTC is the right shape here. A generative model gives you a beam of guesses; if the correct move isn't in the top 8, it's gone. A CTC model emits a probability distribution per character position, so you can compute the exact likelihood of every one of the ~2,000 legal move spellings and rank them. Nothing is ever unreachable.

The metric was lying to me

For two days every experiment returned 23%. Different beam widths, different candidate counts: 23%.

I was measuring "moves correct and in order". The line derailed at half-move 1 and everything after it was a different game, so the number could not move. It also treats a mistake on move 2 and a mistake on move 60 as equally bad, which is nothing like the experience of fixing one.

The metric that turned out to mean something: how many times does a human have to intervene? Simulate the real loop — fix the first wrong move, re-derive the rest, count. That number moves when the model gets better, and it maps directly onto whether anyone would use the thing.

Three measurement bugs, all of which flattered or buried the result

The crop was eating ink. My cell-cropping step discarded connected components shorter than 15% of the cell height, as noise. It was eating the 3 in cxb3. Those half-moves failed in every measurement I had taken. Now it filters by position (the row's band) instead of by size. The rule I should have started with: cropping too much destroys information, cropping too little keeps it even if it's ugly.

A batch truncation. A preprocessing call returned only the first element, so I was evaluating 16 of 64 cells and dividing by 64. The pipeline said 94%, my evaluation said 25%, and I believed the evaluation.

I measured an autoregressive model with a CTC loss. parseq isn't CTC. That "6/24" was my method, not the model.

None of these were subtle in hindsight. All three survived because the number they produced was plausible.

Constrained decoding, and a result I didn't expect

The obvious idea: don't let the model write whatever it wants and clean it up afterwards. Forbid it from emitting any character that doesn't continue a valid move — a trie of the ~2,000 legal spellings, passed to the decoder.

Measured on held-out cells, same model, no fine-tuning:

It costs nothing — same forward pass. It works because the lexicon rescues after the model has already committed, and from types there is nowhere to go. Constrained, the beam spends its branches on the real ambiguity (Bf4 vs Bf7) instead of on choosing dictionary words.

Then the surprise. On my own scoresheets, the constraint made things worse (recall 87% → 70%). Both approaches were about equally good overall — 150 corrections free, 151 constrained — but combining them dropped it to 118. A 21% improvement from two methods that individually tie.

They fail independently. The lexicon proposes by shape similarity (an edit distance that knows which glyphs get confused). The constraint proposes by model probability. Averaging the two normalised confidences beats either.

Three silent holes in my move vocabulary turned up while building this: promotions were missing entirely (one game had d1=Q and a8=Q, unreadable by construction); long castling O-O-O is a single token, not an extension of O-O, so it lives on a different branch of the trie; and Ra1 is a rook in English and a king in Spanish, same characters — the table was silently picking one.

The paper is curved, not rotated

One game accounted for 33 of the 57 remaining corrections. Both causes came from assuming a photograph of a sheet of paper is flat.

The row lines drift between vertical bands from −33 to +14 pixels, and not monotonically — so a global deskew (mine computed 0.26°) fixes nothing. Projected across 3,072 pixels, a line that doesn't coincide with itself smears out and never crosses the detection threshold. The rows were in the photo, perfectly legible. Fixed by letting each vertical band vote with its own maximum and requiring two to agree.

And perspective compresses the lower rows: heights 111, 110, 87, 78, 71. My filter required ±22% of the median and threw away move 25. Now a row only has to resemble its neighbour, not the median — a header jumps abruptly, perspective doesn't.

That game went from 33 corrections to 15, and its recall from 74% to 91%.

The result worth keeping: the user's own games are the dataset

Every cell is already anchored to its half-move number, and a confirmed game gives the true SAN of every cell. So (cell image, correct move) pairs fall out of the pipeline for free, with nobody labelling anything. Four of my own games produced 343 of them.

Fine-tuning on 279 of those and testing on a game that appears in no training set — split by game, not by cell, so the model has never seen that page:

279 of my own cells beat 13,045 borrowed ones, decisively. Recall matches the LLM.

And HCS — a public dataset of handwritten English scoresheets — does something other than what I assumed. On its own it makes things worse: it specialises the model in English notation, where the piece letters are different. But used as a stage before the Spanish fine-tune, it gets you to 98.4% instead of 96.9% going direct. It contributes handwriting; the alphabet has to be re-taught afterwards.

The honest total

Four models, each trained without one game, each scoresheet read by the model that never saw it:

118 → 57 → 36 in one working session, which felt like progress. It is still twelve times the LLM's error count. And that 36 measures a model reading my handwriting, on my club's template, in Spanish. Every one of those four games shares a pen and a hand.

Why I stopped, and it isn't a technical reason

What the recogniser needs is cells from many hands. That corpus comes from users confirming games — which means the line is blocked by exactly the same thing the business is blocked by.

And the cost I set out to remove? At current volume it's close to zero. I spent two weeks optimising an expense that doesn't exist yet.

The right time to come back is when there's a corpus of confirmed games from a lot of different people — which is precisely the moment the saving would start to matter. Parked, not abandoned: 33 commits on a branch that isn't going anywhere.

What I'd tell myself at the start

• Justify the pretrained model you pick. I chose TrOCR by inertia and lost a week to a language model that wanted to read prose.
• Pick a metric that can move, and that resembles what the user experiences.
• When a number looks wrong, look at the input, not the parameters. All three of my measurement bugs were found by opening an image, not by tuning anything.
• Split by the thing you're generalising over. Splitting by cell when the sheets share a handwriting measures memorisation.
• The cheapest labelled data is the data your product already produces.