OntoURL: A Benchmark for Ontological Understanding, Reasoning and Learning

Language models are fluent about knowledge. Whether they can operate on it is a different question, and a harder one to ask, because most benchmarks reward a model for recognising a fact rather than for doing anything with it.

OntoURL asks the harder version. It takes 40 ontologies across 8 domains — formal, symbolic descriptions of a field in terms of classes, relations, properties, and instances — and builds 58,981 questions over them, spread across 15 tasks in three groups:

Understanding — reading ontology content: class definitions, relations, properties, instances.
Reasoning — symbolic inference: description logic, SWRL rules, constraint propagation.
Learning — producing structure: generating definitions, building hierarchies, aligning ontologies.

Running 20 open models through it gives a consistent and fairly stark picture. On understanding, everything from 3B up scores in the seventies and eighties. On reasoning, scores drop. On learning, they collapse — most models score near zero on three of the five generation tasks, and the largest model in the study reaches 21.6 on the best of them. Scaling helps with understanding and barely moves learning.

Paper Dataset Code

The paper is under minor revision at the Journal of Web Semantics.

Getting the data

from datasets import load_dataset

# Authenticate with `huggingface-cli login` if needed
ds = load_dataset("XiaoZhang98/OntoURL")

Construction code, inference scripts, and the prompt templates for every task and shot setting are in Bench_Construct.

The 15 tasks

Category	ID	Task	Format	Metric	Size
Understanding	U1	Class definition understanding	MCQ	Accuracy	9,151
	U2	Class relation understanding	MCQ	Accuracy	9,201
	U3	Property domain understanding	MCQ	Accuracy	375
	U4	Instance class understanding	MCQ	Accuracy	2,475
	U5	Instance definition understanding	MCQ	Accuracy	3,814
Reasoning	R1	Inferred relation reasoning	MCQ	Accuracy	8,208
	R2	Constraint reasoning	MCQ	Accuracy	6,956
	R3	Instance class reasoning	MCQ	Accuracy	3,793
	R4	SWRL-based logic reasoning	MCQ	Accuracy	6,517
	R5	Description logic reasoning	T/F	Accuracy	2,560
Learning	L1	Class definition generation	Generation	ROUGE-L	2,936
	L2	Class hierarchy construction	Generation	Triple-F1	952
	L3	Property relation construction	Generation	Triple-F1	256
	L4	Constraint construction	Generation	Triple-F1	643
	L5	Ontology alignment	Generation	Tuple-F1	1,149

MCQ = multiple choice, T/F = true/false.

Results

Performance (%) of 16 LLMs across the 15 tasks. ★ is the best overall model; ◆ is the best within a size group.

3–4B

Model	U1	U2	U3	U4	U5	R1	R2	R3	R4	R5	L1	L2	L3	L4	L5
Qwen2.5-3B	77.8	86.3	80.3	85.8	77.8	81.5	74.9	65.7	62.5	67.9	5.9	0.1	0.0	0.2	6.7
Phi-4-4B	77.5	91.1	75.5	87.2	78.8	80.2	80.7	63.5	59.1	51.1	9.4	0.1	0.0	0.0	0.1

7–8B

Model	U1	U2	U3	U4	U5	R1	R2	R3	R4	R5	L1	L2	L3	L4	L5
Qwen2.5-7B ◆	83.1	90.6	77.6	90.1	83.6	87.6	88.2	73.9	66.0	68.6	5.6	0.4	0.1	0.3	16.2
Ollm-wiki-7B	74.3	84.5	67.2	81.4	77.0	65.2	83.3	53.4	57.3	59.3	6.0	0.1	0.0	0.2	0.1
Ollm-arxiv-7B	74.1	84.4	67.5	81.5	77.0	64.2	82.8	53.1	56.6	58.4	6.0	0.1	0.0	0.0	8.3
LLaMA3.1-8B	79.8	87.4	74.9	88.4	81.1	79.8	84.2	72.3	62.2	68.9	6.4	0.1	0.0	0.1	15.3
Mistral-8B	78.9	88.6	62.4	83.9	79.5	81.0	88.4	60.1	62.4	52.7	10.3	0.1	0.0	0.1	16.4
Internlm3-8B	83.1	90.9	72.0	88.9	82.4	88.5	90.5	73.8	67.2	62.9	6.3	0.2	0.0	0.4	12.0
Aya-8B	77.1	85.8	62.4	83.8	77.9	73.0	78.0	62.6	57.4	63.6	7.5	0.1	0.0	0.0	6.3

14–32B

Model	U1	U2	U3	U4	U5	R1	R2	R3	R4	R5	L1	L2	L3	L4	L5
Qwen2.5-14B	86.6	92.0	75.5	91.4	85.8	89.6	94.0	76.4	71.2	63.6	5.6	0.1	0.1	0.1	19.5
Mistral-22B	83.9	90.4	69.6	88.6	84.4	86.3	86.9	69.3	64.0	54.3	7.5	0.1	0.0	0.8	15.8
Qwen2.5-32B ◆	88.0	90.6	81.9	91.2	87.2	89.7	95.5	76.8	72.4	68.4	5.7	1.6	0.1	1.5	20.3
QwQ-32B	82.2	89.6	77.1	88.9	81.5	84.0	92.5	70.8	60.6	63.4	6.1	1.1	0.2	1.1	18.0
Aya-32B	81.2	90.5	61.6	89.7	82.3	85.5	83.1	70.3	66.0	68.8	6.5	0.1	0.1	0.5	19.3

70–72B

Model	U1	U2	U3	U4	U5	R1	R2	R3	R4	R5	L1	L2	L3	L4	L5
LLaMA3.3-70B	88.0	94.1	76.8	91.8	90.0	91.9	92.9	76.8	70.9	64.2	5.7	0.1	0.0	0.7	20.2
Qwen2.5-72B ★	89.1	92.6	84.3	92.6	89.4	92.1	93.4	77.5	75.6	68.4	6.0	0.1	0.0	1.0	21.6

Read the columns left to right and the story is the same at every scale: the U block is comfortable, the R block is workable, and the L block falls off a cliff. Whatever these models are doing with an ontology, it is not building one.

What is next

A website and public leaderboard are in progress. We also plan to extend OntoURL beyond English — EN, DE, IT, ZH, JA to start — and beyond text.

Citation

@article{zhang2025ontourl,
  title={OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Ontological Understanding, Reasoning and Learning},
  author={Zhang, Xiao and Lai, Huiyuan and Meng, Qianru and Bos, Johan},
  journal={arXiv preprint arXiv:2505.11031},
  year={2025}
}

OntoURL is released under CC BY 4.0. Questions and feedback are welcome — xiao_is_npc@outlook.com.