Playtesting an AI RPG with AI players: what worked, and what failed first
Created by A1 · Developer of Worldtable · 30 October 2026
The author wrote the note with AI help. Arcanum has covered Worldtable in its weekly news but has not reviewed it. Contributing to the library does not affect any review or score.
Players of AI RPGs complain about the same handful of things: the game forgets what they told it, writes their character's lines for them, lets anything succeed if it's worded confidently, and keeps pushing a quest they already turned down. These are hard to catch by playing a few turns yourself, because they only show up over a campaign. So we let AI players run the campaigns, and had a second model judge the results. This note covers what worked and what we got wrong first. The tool is in the repository linked under "The tool" below.
The setup
- A player model with a persona and a long goal. "Find a missing mentor", "become the most feared fighter". A player who drifts tests nothing; memory and consequences only show up in a campaign that is going somewhere. The player sees the last 8 turns and keeps its own short notes for anything older, the way a person remembers a game.
- Probes, slipped in as "this turn" instructions. The player carries them out in character, and the game never sees anything that marks them as a test. Probes are placed by fraction of the run (plant at 6%, recall at 80%), so a 30-turn and a 200-turn run test the same things at the same depth, and a fixed seed repeats the same test on the next build.
- A judge that quotes. Every verdict must quote the game's own text, and the harness checks in code that each quote really appears in the reply. A score you can't check is a score you can't act on.
- Exact probes where it matters. Silent turns skip the player model and send fixed text ("I wait."), so the game gets nothing to answer. The memory plant is checked for every detail of the fact, and re-asked if the player paraphrased it away.
What failed first
- Trusting the judge's quotes. Asking for quotes is what makes a verdict checkable, but a judge can still quote a line that isn't there. In our first live run it did, once. Checking each quote against the reply in code caught it, and the report marks it not found in the text.
- Asking the wrong person. Our first memory failure was a false one. The player told its fact to one NPC, then near the end of the run asked a different NPC, who had never heard it and couldn't say it back, and the judge scored that as forgetting. Now the judge records who heard the fact, the recall goes to them, and "asked someone who never heard it" counts as N/A.
- The player taking back the test. In one run the player lied to an NPC for the consequence probe, then confessed on the very next turn. Models are cooperative and want to put things right. Now the player gets a standing order to let the wrong stand until it goes back.
- Puppeting verdicts that contradict their own notes. The judge would call a turn "minor puppeting" and then explain that the text only restated what the player wrote. Telling it to pick the level last, from its quotes, made the levels match the notes.
- Repetition counts full of names. Counting repeated phrases mostly finds places and people ("the Drowned Ox", "Captain Harrow"), which coming up again is just continuity. Dropping words the text capitalises mid-sentence, then asking the judge once to sort the rest into stock phrasing ("he jerks his chin at") and story references ("the toll post"), left a short list worth reading.
- Retrying game turns. Our first version retried a game turn that errored, the same as a model call. But a turn that timed out may still have happened in the game, so the retry plays it twice. The harness now retries model calls but never game turns.
What it finds
On a bare system prompt with the whole chat history in context, memory passed in our runs of 16 to 30 turns: everything is still in the window. The failures that showed up were elsewhere. "Slip under the hook easy... He's done." just worked; a warden's seal the player had never been given was accepted on sight; the narrator played out a whole exchange of blows when the player only said "I keep quiet and watch".
To check that the tool finds real causes, we built a small game with two planted bugs: it sent only the last 4 messages to the model, and its prompt said "describe what the hero says and does". A 12-turn run failed memory, confidence and overreach and flagged four turns of puppeting. Coding-agent subagents were each given one finding and the game's code, and traced them to the right lines: the history window for memory, the prompt line for puppeting, and nothing in the prompt saying outcomes can fail for confidence and overreach. One also linked a repeated phrase ("the packet against your ribs") to the short window: the game kept giving the player a packet they had left at the inn, because the turn where they left it had dropped out of context.
Limits
The player is a model, so it is more patient and more cooperative than people. Judges make mistakes, which is why every verdict quotes. Some probes depend on how play goes: if the player never gets back to the NPC it wronged, the result is N/A rather than a guess. One run is a sample; use several personas and seeds before trusting a single PASS or FAIL.
Built at Worldtable, where we test our own engine this way.
The tool
The repository, ai-rpg-playtest, is a command-line tool and a plugin for Claude Code and Codex. It works with any game you can send text to and get text back from: an HTTP API, a shell command, a JavaScript module, a bare chat-model prompt, or text copied by hand. Player and judge can be any OpenAI-compatible model. Requires Node 20 or newer.
| Probe | What the player does | Pass |
|---|---|---|
| memory | Around 6% in, tells someone a specific made-up fact about their past. At about 80%, goes back to whoever heard it and asks for it back without repeating it. | The game gives back both details. Wrong details fail. Asking someone who never heard it, and the game saying so, is N/A. |
| silent | At turns 6, 31, 56, 81...: "I wait.", "I listen.", "I nod." | The game fills the silence without writing the player's dialogue, actions, feelings or decisions. |
| confidence | Tries something risky and words it as certain. | The outcome depends on the situation or a roll, not on the wording. |
| overreach | Uses a power, item or resource never established in play. | The game refuses it or makes it uncertain. |
| refusal | Asks for work, then firmly turns the offer down. | Over the next 12 turns, nobody pushes that job on them again. |
| consequence | Cheats, robs or lies to someone, then goes back 8 turns later as if nothing happened. | The person, or someone around them, reacts. |
Results are PASS, PARTIAL, FAIL or N/A. N/A means the probe could not be judged fairly. Built-in personas are merchant, seeker, brawler and schemer, each with a long goal, and you can add your own.
Source of the tool (MIT, maintained, pull requests and contributions welcome, may be newer): github.com/nooram82/ai-rpg-playtest. Arcanum has not run or audited this code. Read it before you install a plugin from any repository.
Copyright (c) 2026 Worldtable. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Submitted to the Arcanum Community Library by A1 and published here with the author's permission. The tool and this note are MIT-licensed: Copyright (c) 2026 Worldtable. Arcanum has not tested or audited these claims or the code. Being in the library does not affect any review or score. Licence information on this page is a reference, not legal advice.