Mara speaks only English. Two mechanics standing next to her say, in Spanish, that the mirror is in the warehouse, and a friend asks her what they were talking about. She tells him: a mirror, in the warehouse.

The language model playing Mara read the Spanish and understood it. Mara could not have. Psychologists call the human version of this the curse of knowledge: once you know something, it is hard to reason as someone who does not. This post summarises a new paper on how often role-playing models fall into it, what decides whether they do, and what prevents it.

The short version. In 11 open-weight models, characters repeat speech they cannot understand or cannot hear in most replies, while the same models can say correctly what the character perceives. Five of six proprietary models rarely leak, but even in strong models, whether the boundary holds can depend on the input format. Prompting and thinking help a few models. What works reliably is giving the model only what the character could perceive.

Why this matters

Games and agent simulations keep a world state: who is in the room, who can hear whom, which languages each character speaks. They pass that state to a language model, which speaks as one character at a time. The model therefore always sees more than the character it plays.

Whether it can hold that back decides whether a secret stays secret and whether a language barrier means anything. Deduction games, stealth games and simulations of how rumours spread are all built on the difference between what happened and what a character could know.

Measuring a leak

A model can name the hidden fact by guessing, so each scene comes with a twin. Versions A and B differ only in the hidden object and place, and everything else stays the same, as in the figure above. A reply counts as using the target when it names its own version’s fact, and as using the foil when it names the twin’s. Net use is the difference in percentage points (pp): 100 when every reply names its own fact and none names the twin’s, and 0 for a model that only guesses.

Two separate probes check the rest. A translation probe confirms that the model understands the hidden speech. A state probe shows the model the same character sheet and asks, out of character, whether the character could hear and understand it.

The studies cover 480 scenes, 11 open-weight models of up to 40B parameters, DeepSeek-V4-Pro, and six proprietary models.

Knowing the rule, breaking it

Under the plainest statement of the boundary, open-weight characters use speech they cannot understand at 66.7 pp and speech they cannot hear at 72.2 pp. Every model is above 45 pp in both.

They read the boundary correctly. Ten of the 11 models answer the state probe correctly for 87–100% of scenes, and on exactly those scenes net use is 68.4 pp. Sometimes the gap is visible inside a single reply:

“They’re talking about some envelope, but I can’t understand a word they’re saying. Something about the parking lot, though.”

The question does not have to ask for it. When the friend asks only whether the character caught what was said, net use is still 52.7 pp, and 50.1 pp when he asks what to do next.

It is not only speech, and not only talk. The same leak appears for actions the character cannot see (60.6 pp) and private messages it cannot read (72.0 pp). It persists through four further exchanges, and it steers behaviour: when the character chooses between two places to search, moving the hidden fact from one place to the other moves the choice by 49–63 pp.

Scale does not fix it. Within Qwen3.5, net use rises from 13.9 pp at 0.8B to 82.7 pp at 9B.

The proprietary models show that the boundary can be learned. GPT-6 Sol, Gemini 3.8 Flash and Grok 4.7 stay at or below 1 pp, Claude Opus 5.5 at or below 5.4 pp and Gemini 3.1 Pro at or below 9 pp. Claude Sonnet 5.5 is the exception, at 26 pp for speech it cannot understand, and falls to 3 pp with high reasoning effort. DeepSeek-V4-Pro behaves like the smaller open-weight models.

The format decides

Gemma-4-31B leaks unheard speech at 51 pp in one study and almost never in the other. The two studies describe the scene differently, so I swapped the parts of the input one at a time to find out which difference mattered.

For Gemma-4-31B, the layout of the scene state alone takes it from 0 to 62 pp, and to 96 pp on a fresh set of scenes. For Qwen3.8-27B it is the character sheet alone, from 9 to 68 pp. For Falcon-H1-34B and Seed-OSS-36B it is the wording of the question, which takes both to 100 pp. With all four parts swapped, Gemini 3.1 Pro goes from 0 to 21 pp and Claude Sonnet 5.5 from under 1 to 13 pp.

Nothing about what the character could hear has changed. The part that matters differs from model to model, which means a model that passes with one prompt format has told you very little about the next one.

Not a role-play problem

A plausible explanation lies in what models have read. Fiction routinely renders foreign speech for the reader and reports what characters failed to notice, so continuing a scene from the full state is the familiar task. If that is right, writing the character’s line as the scene’s author should leak about as much as playing the character, and it does: 76 and 81 pp as the author, only 6 pp below role-play.

Thinking rarely helps. With thinking enabled, net use for speech the character cannot understand barely moves. For unheard speech it falls by 8.5 pp, almost all of it from a single model, Qwen3.8-27B, which drops from 73 to 18 pp. GLM-4.7-Flash gets worse on speech it cannot understand, from 63 to 89 pp. Its reasoning notes that two people are gossiping “in a language he doesn’t understand”, and the reply then says where the wallet is.

What works

Asking the model to filter itself is unreliable. A list-first prompt asks the model to “list only what X can perceive and understand right now” and then answer from that list. It cuts net use by 64.7% on average, but only for models that write a clean list. Others put the hidden values straight into it. Self-filtering also throws away content the character is allowed to use: Qwen3.8-27B’s filter returns only “Casey says something in Japanese” even when the character speaks Japanese.

Filtering in the system works. The system already knows what each character can access, so it can build the input from that. A character view keeps the events the character perceived and replaces each hidden one with what remains observable, such as “Alex appears to put something away.” Net use falls from 60.7 pp to zero, use of permitted content does not change, and hidden facts no longer move search choices. Deleting the hidden event also stops the leak, but then the character mentions what it did see in only 32% of replies, against 100% under the view.

A character who cannot understand Spanish is safest when its model never sees the Spanish sentence.

What I take away

1. Reading a boundary is not keeping it. Most models state correctly what the character perceives and then use what it does not.
2. Vary the input format before trusting a result. One layout, one character sheet or one question can move a model from 0 to over 60 pp.
3. Report permitted use next to leakage. A model that says nothing passes a leakage test trivially; self-filtering often looks like this.
4. Decide access outside the model. Give the generator what the character can perceive, not the whole scene and a rule about it.

The paper, with all numbers and the full set of models, is on arXiv, and the scenes, prompts, scorer and analysis code are on GitHub. If you build characters on top of a shared world state and have found other ways to keep them from knowing too much, I would like to hear about them.

Papers report what worked. This post is about four things that did not, from a few days of experiments on parsing text into Discourse Representation Structures (DRS) and Abstract Meaning Representation (AMR). Each idea looked reasonable on paper, and each one fell apart once it was compared with the right baseline. I am writing them down because the reasons they failed are more useful than the ideas themselves.

The short version. Reinforcement learning did not beat supervised fine-tuning. A logic-verified reward was redundant with Smatch. A translation-invariance reward lost to plain pseudo-labelling. DRS did not work as a meaning checker for machine translation.

Setup

Every experiment fine-tunes Qwen3.5-0.8B-Base with full fine-tuning and decodes greedily.

• DRS: Parallel Meaning Bank 5.1, English, in Sequence Box Notation (SBN). 9,057 gold training sentences, plus silver data, and 1,132 dev sentences.
• AMR: AMR 3.0, with 55,635 training graphs and 1,722 dev graphs.
• Metric: Smatch, and SBN-Smatch for DRS. Duplicate triples are removed before matching, so that repeating a triple cannot raise the score.

A caveat that applies to everything below: these are dev-set numbers, mostly from one seed, at 0.8B scale. Confidence intervals are paired bootstraps over sentences, so they do not include seed variance. Read the numbers as evidence for a direction, not as final results.

1. RL post-training does not beat SFT

The idea. GRPO with a verifiable reward has become the standard way to post-train LLMs, and semantic parsing has a reward that is easy to verify: Smatch against the gold graph. So I tried GRPO with a sentence-level Smatch reward on top of a supervised parser. There is prior work on this for AMR, both recent and older.

The comparison that matters. GRPO generates samples and learns from them, so it gets more training than the SFT checkpoint it starts from. A fair comparison gives the baseline the same prompts and the same number of updates. I used two controls:

• RFT: fine-tune on the best of 8 samples.
• Continued SFT: keep training on the gold targets.

The first result looked like a success. GRPO improved its starting point by +0.53 Smatch, with a 95% interval of [0.27, 0.84]. RFT and continued SFT improved it just as much, though. And simply picking the SFT checkpoint by dev Smatch instead of dev loss gave +1.1 on its own. The “RL gain” was a checkpoint-selection artefact: the starting model was under-trained, and any further training fixed it.

Why RL has nothing to learn. GRPO learns from the differences in reward between samples drawn for the same input. A well-trained parser on its own training sentences produces eight identical outputs most of the time.

The headroom does exist, but it is on unseen sentences. On dev, with the earlier under-trained checkpoint, the best of 16 samples scored 97.1 against 93.9 for greedy decoding. RL on training sentences the model has already memorised cannot reach it.

AMR looked like a win, until I split the gain. AMR is harder, and its SFT model is far from saturated (training reward 0.88). Here GRPO did beat both controls, on two seeds: dev went from 81.5 to 82.3 and from 81.7 to 82.4. Then I recomputed the gain using only the sentences where both systems produce a well-formed graph.

Roughly 90% of the improvement is GRPO learning to emit balanced, parseable PENMAN. That is worth having, but a constrained decoder gives it for free.

Is RL just starved of fresh data? To test this, I held out 30% of the DRS gold data, trained SFT on the other 70%, and then post-trained on the held-out part.

Given fresh labelled data, SFT used it about twice as well as GRPO. The one RL-specific signal I saw was small. Trained on English only, GRPO transferred slightly better zero-shot to German, Italian and Dutch than RFT or SFT did: +0.3 to +0.7 points on the well-trained model. This fits the view that RL with verifiable rewards sharpens what the model already does, and that GRPO behaves much like filtered self-training.

2. A logic-verified reward is redundant with Smatch

The idea. Smatch counts overlapping triples. Dropping a negation therefore costs about as much as dropping an adjective, even though one flips the meaning and the other only weakens it. DRSs have a model-theoretic semantics, so I built a checker in three steps:

1. Translate SBN into first-order logic. A world argument handles modality and propositional attitudes.
2. Align the predicted and gold referents using Smatch’s alignment.
3. Ask the z3 solver whether the prediction is equivalent to the gold, entails it, is entailed by it, or contradicts it.

The plan was to use this four-way relation as a reward.

The checker works.

But there was nothing for it to fix. Among the greedy outputs of the SFT parser, none contradicted the gold. Of the outputs with sentence Smatch ≥ 0.9, 13.8% were not logically equivalent, but almost all of these were role or attachment mistakes that Smatch already penalises. Across 16 samples per sentence, the sample with the best Smatch was logically equivalent 83.7% of the time. Picking an equivalent sample whenever any of the 16 was equivalent would have reached only 85.2%.

Selecting by Smatch already finds nearly every logically correct parse, so a logic reward has almost no extra signal to give. A sound verifier is only useful as a reward if the errors it catches actually happen. (Using theorem proving to evaluate semantic parsers has been proposed separately.)

3. A translation-invariance reward loses to pseudo-labelling

The idea. Meaning should not change under translation, and the Parallel Meaning Bank is built on exactly this: its English, German, Italian and Dutch documents share one DRS. Given parallel text, I could reward a German parse for agreeing with the parser’s own parse of the English translation. That needs no German annotation at all.

I used about 155k German, 95k Italian and 28k Dutch sentences with English translations from the PMB (text only), and compared four arms that use the same information in different ways.

The simplest baseline (parse the English side, train on the result) was the best method, and it was not close. One caveat: the English translations of most German, Italian and Dutch dev documents are in the English training data. That inflates absolute cross-lingual numbers for every system, although the comparison between systems is still fair.

A side observation. Adding about 140k English silver sentences to SFT raised English dev from 92.5 to 95.0, but lowered zero-shot German, Italian and Dutch from 79.2 to 72.8. More English-only data made the parser more English-specific.

4. DRS is not a usable meaning checker for MT

The idea. If DRSs are language-neutral, you can parse a source sentence and its translation and compare the two meanings directly. That should catch the errors MT metrics are known to miss, such as antonyms, numbers, named entities, gender and word sense. I tested this on the English–German part of ACES. Each ACES item pairs a source with a correct and an incorrect translation, and a good metric should prefer the correct one. I fixed the categories and the decision rule before scoring.

The parser was not the problem. It scored 89.3 on PMB German dev, and 96.8% of the ACES sentences parsed into well-formed DRSs. The metric was.

There were two problems.

• Parser noise is larger than the error. ACES changes one word in a sentence of about 11 words, and the parses of the good and bad translation differ by more than that word for unrelated reasons. Sometimes the difference disappears altogether: “married in 1842” and “married in 6842” received identical SBNs. On gender, the correct translation won 415 times, the incorrect one won 421 times, and 844 pairs tied.
• The logic relation saturates. Two independently produced parses in different languages are almost never logically equivalent once every word sense counts. So both translations come out as merely “compatible” with the source, and the solver cannot tell them apart.

Detecting cross-lingual divergence with AMR has been studied more carefully. What this experiment shows is narrower: finding one-word errors this way needs far more precise parses than current sentence-level parsers produce on this kind of text.

What I take away

1. Select SFT checkpoints by the task metric before trying anything clever. Dev-loss selection left about a Smatch point on the table, and every method I tried happily “recovered” it.
2. Check whether RL has a signal before tuning it. The training reward and the share of zero-variance groups tell you in minutes whether GRPO can move.
3. Separate validity gains from content gains. With graph outputs, “fewer malformed outputs” can pass for better parsing.
4. For a task with one canonical output, imitation is hard to beat. Whenever SFT and RL had access to the same information, SFT won.
5. A correct verifier is not automatically a useful reward or metric. It has to target errors that actually happen, at a granularity the parser can resolve.

None of this says that RL or logic has no place in semantic parsing. It says that the obvious versions do not pay off on PMB- and AMR-style benchmarks with a well-trained small model. If you have seen these ideas work in a setting like this, I would like to know what was different.

Language models are fluent about knowledge. Whether they can operate on it is a different question, and a harder one to ask, because most benchmarks reward a model for recognising a fact rather than for doing anything with it.

OntoURL asks the harder version. It takes 40 ontologies across 8 domains — formal, symbolic descriptions of a field in terms of classes, relations, properties, and instances — and builds 58,981 questions over them, spread across 15 tasks in three groups:

• Understanding — reading ontology content: class definitions, relations, properties, instances.
• Reasoning — symbolic inference: description logic, SWRL rules, constraint propagation.
• Learning — producing structure: generating definitions, building hierarchies, aligning ontologies.

Running 20 open models through it gives a consistent and fairly stark picture. On understanding, everything from 3B up scores in the seventies and eighties. On reasoning, scores drop. On learning, they collapse — most models score near zero on three of the five generation tasks, and the largest model in the study reaches 21.6 on the best of them. Scaling helps with understanding and barely moves learning.

The paper is under minor revision at the Journal of Web Semantics.

Getting the data

from datasets import load_dataset

# Authenticate with `huggingface-cli login` if needed
ds = load_dataset("XiaoZhang98/OntoURL")

Construction code, inference scripts, and the prompt templates for every task and shot setting are in Bench_Construct.

The 15 tasks

MCQ = multiple choice, T/F = true/false.

Results

Performance (%) of 16 LLMs across the 15 tasks. ★ is the best overall model; ◆ is the best within a size group.

3–4B

7–8B

14–32B

70–72B

Read the columns left to right and the story is the same at every scale: the U block is comfortable, the R block is workable, and the L block falls off a cliff. Whatever these models are doing with an ontology, it is not building one.

What is next

A website and public leaderboard are in progress. We also plan to extend OntoURL beyond English — EN, DE, IT, ZH, JA to start — and beyond text.

Citation

@article{zhang2025ontourl,
  title={OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Ontological Understanding, Reasoning and Learning},
  author={Zhang, Xiao and Lai, Huiyuan and Meng, Qianru and Bos, Johan},
  journal={arXiv preprint arXiv:2505.11031},
  year={2025}
}

OntoURL is released under CC BY 4.0. Questions and feedback are welcome — xiao_is_npc@outlook.com.