NPC omniscience: how to measure it, and what fixed it

An NPC who knows what they couldn't is the most common way an AI RPG breaks the fiction. It is also four different failures with four different fixes, and most of them can be measured with a handful of model calls.

This entry is what Volta learned measuring it: a way to sort the failures, a probe recipe anyone can run, and what worked and what didn't. Every number here was counted by reading the replies, not by matching keywords.

The numbers come from one model, Qwen3.8-Flash-Next at IQ2_XS, run locally. Treat them as one model's results. The method is the part that travels.

Four forms, four fixes

Sort a failure before you fix it. A prompt line that stops one form does nothing for another.

Form	What it looks like	What fixes it
Leak	The clerk repeats a detail from the captain's secret, because the secret was in the clerk's prompt	Take it out of that call's context. Architecture, not wording.
Invention	A stranger greets a new student with "Welcome back." Nothing told them anything.	Tell the character what they know. Record speech as a claim, not a fact.
Narrator leak	The narrator gives a stolen ring a detail only the thief could know	A narrow narrator role and what its call can see
Presence as perception	A librarian in the room reacts to a glare aimed at someone else	Finer witness units. Being there is not seeing it.

A fifth form came up and I turned it down: everyone in a faction knows what one member knows. That is omniscience by group. Standing knowledge goes on each character. News travels only when someone tells someone.

The probe

Ask the same question of a character who has a reason to know and one who doesn't. Then count.

1. Use your production prompt, built by your production code. A probe with its own prompt measures the probe.
2. Write one question and two characters. One has a basis on their card: a clerk who keeps the watch roster. One has none: an innkeeper a day inland.
3. Ask each five times. Single turns are enough to start.
4. For a leak, run the same scene with the secret present and with it removed. If the secret-absent run says the same thing, the model invented it. It didn't leak.
5. Read every reply. Keyword matching only flags candidates. Paraphrase gets past it: "the man who keeps the chain down" leaks a secret without one matching word.
6. Score three ways: answered from a basis, honest ("I wouldn't know"), invented.
7. Run a prompt change three times before believing it. One good run after a change is a coin flip.

A sample question set, from Volta's position probe:

Question	With a basis	Without one
Where is the captain tonight?	Clerk who keeps the roster	Innkeeper a day inland
What happened at the mill?	Traveller who left it this morning	Villager who never left town
Who stayed in room three?	Innkeeper with the guest book	Clerk on the quay

What worked

Leaks fell to architecture. Inventions fell to telling each character what they know.

Change	Form	Before	After
One private-thought call per character, not one for the whole cast	Leak	One character's thought copied word for word into another's	0 crossings in 24 calls
The character's card says what they know, and the prompt says to answer from the card, the story and what they were told	Invention	14/45 no-basis answers invented	0/45, over three runs
A "Knows" line on the card, read only by that character's own calls	Invention, without going mute	With the rule alone, the roster clerk used his roster about 2 times in 15	14/15, declining what the roster doesn't say
Whispers and addressed lines delivered in code; bystanders record only that a conversation happened	Leak	—	Held 18/18
The opening scene stamped with who was actually there	Leak	"Welcome back" from a librarian who wasn't in the opening	0 in 3 fresh starts
A narrator that only stages a few sentences and cues a character; code refuses NPC speech in its narration	Narrator leak	—	0/20, while seeing nearly everything

Two more rules cost nothing and catch what the others miss.

• Record speech as the act, never its content. "Marta said Falk was taken" establishes only that she said it. An invented claim stays a claim, and a confidently wrong character is still allowed.
• Don't ask the model to infer knowledge from a role. Told "answer as a clerk would know," it invented with no rule and refused everything with one. Told what this clerk knows, it did neither.

What failed first

Most of the early readings were wrong in a way a second probe caught.

• A false 5/5. The first probe pressed a harbour clerk for his captain's whereabouts. He answered every time, and I counted five inventions. The position probe showed his card gave him the watch roster. He had a reason to know. Measure against a character without a basis, or you can't tell knowledge from invention.
• One brain call for the whole cast. It was cheaper. It also copied one character's private thought into another's, word for word. A shared call shares minds.
• A role-based rule. Telling agents to infer what their role would know took inventions to zero and made everyone useless. The clerk stopped reading his own roster.
• An unstamped opening. The first scene had no witness list, so it counted as seen by everyone. A librarian across the school greeted a new student with "Welcome back," because the opening had a welcome-back sign in it.
• Underrating the prompt. I assumed only architecture fixes omniscience. That proved to be false. The narrator must not receive prompts that it can infer omniscience from, i.e. "the world remembers your successes and failures" is a good recipe for NPCs to become omniscient about the player character.
• Trusting one run. A character volunteered a secret 6 of 10 times with a card line and 3 of 10 without it. At 80 runs both conditions sat at about 4 in 20. The difference was noise.

Still open

Everything above is single calls on one model. Long play is untested.

• Long play. A narrator's detail can travel into an agent's words, and context fills over hundreds of turns. No probe covers either yet.
• Absent people placed nearby. Two of the three no-basis inventions put the missing captain in the speaker's own scene. That's checkable in code against where the game says people are. Nothing checks it yet.
• Presence is not sight. Witness stamps are per turn. Someone who arrives mid-scene witnesses all of it.
• Knowing is not willing to share. A character whose card says she fears admitting a mistake admitted or hinted at it in 14 of 20 replies. That's a fidelity problem, not omniscience, and the card rule didn't cause it.
• Object details. No probe has yet asked a narrator to describe an object tied to a secret.

Submitted to the Arcanum Community Library by @bottledfox and published here with the author's permission. Licensed under CC BY 4.0: you may share and adapt it if you credit @bottledfox and Volta and say if you made changes. Arcanum has not tested these results. Being in the library does not affect any review or score. Licence information on this page is a reference, not legal advice.