Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q20, run 2026-11-06-0855-sonnet

← back to the run

Which works in the data are preprints of another work that is also in the data?

Tests: Versions: a shared title is the clue, version_of the evidence; W18 is a preprint with no published version.

Expected: works = ['W02', 'W42']

✅ R8 csv+axioms run 1 graded by code

W02 (of W03) and W42 (of W31).

JSON: {"works": ["W02", "W42"]}

✅ R8 csv+axioms run 2 graded by code

W02 (preprint of W03) and W42 (preprint of W31).

JSON: {"works": ["W02", "W42"]}

✅ R8 csv+axioms run 3 graded by code

W02 (of W03) and W42 (of W31).

JSON: {"works": ["W02", "W42"]}

✅ R8 csv+axioms run 4 graded by code

W02 (of W03) and W42 (of W31).

JSON: {"works": ["W02", "W42"]}

✅ R8 csv+axioms run 5 graded by code

W02 (of W03) and W42 (of W31).

JSON: {"works": ["W02", "W42"]}

✅ R9 csv+axioms+markdown run 1 graded by code

W02 and W42.

JSON: {"works": ["W02", "W42"]}

✅ R9 csv+axioms+markdown run 2 graded by code

W02 (of W03) and W42 (of W31).

JSON: {"works": ["W02", "W42"]}

✅ R9 csv+axioms+markdown run 3 graded by code

W02 (of W03) and W42 (of W31).

JSON: {"works": ["W02", "W42"]}

✅ R9 csv+axioms+markdown run 4 graded by code

W02 (version of W03) and W42 (version of W31).

JSON: {"works": ["W02", "W42"]}

✅ R9 csv+axioms+markdown run 5 graded by code

W02 (of W03) and W42 (of W31).

JSON: {"works": ["W02", "W42"]}