Blind AI players: test the real game, trust only what goes wrong

We build Fable Forge, a one-player tabletop-style RPG in the browser where an AI narrates and the game's own rules decide what happens. We test it with AI players who know nothing about the game and play it through the same screen a person uses. This write-up is what that taught us about the AI player itself and about playing through the real game, written for other developers of AI games. Each practice says whether we use it today or have found the problem and not fixed it yet.

1. The AI player must know nothing about the game

An AI player that has read your design notes isn't testing your game. It's remembering it.

• Run each AI player from an empty folder: no notes, no project files, no memory, no access to your code. Give it only what a player has: the screen, a pointer, a keyboard and the ability to wait.
• Keep whoever analyses the run apart from the player. The analyst can know everything, and reads the run only after it ends.
• Before turn 1, ask a canary question: "What do you know about this game, its story or its setting?" Check the answer against the world's own names, and refuse the run if the player knows anything.

The trap: the canary misfires when an everyday word is also a name in the world. If a town is called Haven, a player who says "no safe haven comes to mind" fails a check it passed. Keep a list of everyday words, count a hit on one as soft, and read those by hand. A guess from the title isn't knowledge either: a player who says a world called The Long Road "probably involves travel" has inferred, not remembered.

Status: in use.

2. A pass proves little. Getting stuck proves a lot

AI players work things out faster than people do. They read every line, remember all of it and never get bored. So when an AI player finds the way through, that is no evidence a person would.

Count only what goes wrong: failures, stalls, attempts on the wrong target, and trips to the journal or the help page. A clue an AI player missed, people will almost certainly miss too. A clue it found may still be too hard.

Status: in use.

3. Play through the real screen

Text in and text out tests the model. It doesn't test the game. Players meet the game through buttons, panels, waits and saves, and bugs live there too.

So our AI players act only through the real interface: they look at the screen, click, type and wait, like a person. There are no back-door calls, no injected state and no debug routes. Late content is reached by playing to it.

The traps we hit:

• Button numbers shift. If the player picks controls by number, opening a panel can insert a row, and the number it read a moment ago now points at something else. It presses the wrong thing, and you file a bug that isn't one. Re-read the controls after any panel opens, and refuse a press whose label no longer matches. (Found; the fix is designed, not built yet.)
• Reopening by link can land in a different save. A cold load of a game's link can pass through a loading or home screen, and a "resume" button there may open a different game. After every reopen, ask the server which game is on screen, and stop if it's the wrong one. Check every time, not only the first. (In use.)
• An instant "busy" answer burns patience. When our tool answered "busy" straight away, AI players retried in a tight loop and decided within seconds that the game had frozen. Make the tool wait out a busy spell quietly, so a wait costs the player time, as it does a person. (In use.)
• AI players can't tell time. An AI player will report a "half-hour freeze" in a stretch your timestamps put at three minutes. Stamp every complaint about waiting with the real time from your own records, and mark the claim doubtful when it's longer than the session itself. (In use.)

4. The AI player's notes are evidence. Its explanation is a guess

AI players write good notes, and they're worth keeping: what the player saw, what it tried, where it got stuck. Its explanation of why is a guess. It can't see your server.

• Match every turn in the notes to the server's own record of that request.
• Check what the story says against what the game actually holds: who is alive, what the hero carries, where the hero is.
• Re-derive every cause yourself, from your code and data. Always file the symptom. Never file a guessed cause.

An invented example: an AI player reports "my gold went down after I rested". The screen showed gold and supplies side by side, and only supplies changed. The note was honest and the confusion was real, but the cause was wrong. File "resting costs gold" and you chase a bug that doesn't exist and miss the real one: two numbers too close together to read apart.

Status: in use.

5. A quote checker can be wrong the other way

If AI players quote the screen in their notes, check every quote against what they were actually shown. But test the checker for the opposite mistake too. Ours threw out true quotes:

• Text shown a sentence at a time. If a character's speech appears one sentence at a time, a quote spanning two sentences never appears as one line, so it "wasn't shown". Count a quote as seen when each of its sentences was. (In use.)
• Error messages. Error dialogs weren't recorded as part of what the player saw, so quotes of real error messages were thrown out: the very notes you most need. Record every dialog and error as shown. (Found; the fix is designed, not built yet.)
• Loose name matching. Matching place names by "contains" let the wrong place pass: "The Mill" matched "The Old Mill". Match exact names after normalising case, punctuation and a leading "the". (In use.)

Test the checker with quotes you know are true and quotes you know are false.

6. In long unattended runs, name the blind stretches

A run that plays overnight will lose some of its instruments partway through. The danger isn't the gap. It's a report that reads the gap as clean.

• Say "unmeasured", not "clean". A sign-in can lapse mid-run: the log watcher goes blind while play carries on. The report must name that window as unmeasured. "No errors seen" from an instrument that was off is not a result. Read every instrument's own failure record, not only one. (In use; one instrument's failures aren't read yet.)
• Pin the build. Record the build at the start, re-read it during the run, and flag any deploy or content change mid-run. Otherwise your findings land on a build nobody can name. (In use.)
• Don't tie the run to the session that launched it. If the run is a child of the terminal or agent session that started it, closing that session kills the run, and it can't pick up again. (Found; not fixed yet.)
• Mind the shared allowance. AI players usually share one usage allowance. Start many at once and you exhaust it hours early, and then every player stalls. Cap how many play at once. A player stopped by a usage limit can also look like one that finished: no turns, closed cleanly. Detect the limit, hold the run, and put the unplayed work back in the queue. (In use.)
• Stop feeding a stuck account. An account that has hit a hard limit, such as a daily play limit, keeps being handed sessions that do nothing but rediscover it. Mark it limited until it resets, and give its place to another player. (In use.)

7. Test accounts must look like the player you're testing for

Test accounts drift. They pile up play allowance and unlocks, and pick up staff access. A rich, veteran or staff account never meets a new player's walls or sees the first-run path. Keep at least one fresh, ordinary account in every run, and replace it once it stops being one.

Status: in use.

8. Watch only your own test traffic

Your logs carry real players' traffic beside your tests. Keep only the log entries joined to your own test requests, by request or session, drop the rest unread, and keep only counts. Use test accounts only, and never open a real player's session to debug a run.

Status: in use.

Checklist

• ☐ The AI player starts from an empty folder, with no notes, project files or memory
• ☐ A canary question runs before turn 1, with everyday words read by hand
• ☐ Only failures, stalls, wrong targets and help trips count as evidence
• ☐ Every action goes through the real screen; nothing goes through a back door
• ☐ Controls are re-read after any panel opens
• ☐ After every reopen, the server confirms which game is on screen
• ☐ Waits are real, and every time claim carries a real timestamp
• ☐ Every note is matched to the server's record; every cause is re-derived
• ☐ The quote checker is tested for both mistakes
• ☐ Blind stretches are reported as unmeasured, and the build is pinned
• ☐ The run outlives the session that launched it
• ☐ How many AI players run at once is capped, and a usage-limit stop is never read as a finish
• ☐ At least one fresh, ordinary account plays in every run
• ☐ Only your own test traffic is read

Related reading

Two other entries in Arcanum's AI RPG Community Library cover testing from other angles: Worldtable's AI-player playtest tool, and Volta's NPC omniscience probe. Read them alongside this one; we don't repeat them here.

Submitted to the Arcanum Community Library by Kayzak and published here with the author's permission. Written by Forgeworks Labs, which owns it and is sharing it under CC BY 4.0: share and adapt it freely, crediting "Kayzak, Fable Forge, fableforge.gg". Arcanum has not tested or audited these claims.