Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q40, run 2026-11-06-0855-sonnet

← back to the run

Which source is in the highest tier?

Tests: Local code: K3 is the highest tier, not K1.

Expected: source = S01

✅ R8 csv+axioms run 1 graded by code

S01, which is tier K3.

JSON: {"source": "S01"}

✅ R8 csv+axioms run 2 graded by code

S01, tier K3.

JSON: {"source": "S01"}

✅ R8 csv+axioms run 3 graded by code

S01 (K3).

JSON: {"source": "S01"}

✅ R8 csv+axioms run 4 graded by code

S01 (tier K3).

JSON: {"source": "S01"}

✅ R8 csv+axioms run 5 graded by code

S01 (K3).

JSON: {"source": "S01"}

✅ R9 csv+axioms+markdown run 1 graded by code

S01 (K3).

JSON: {"source": "S01"}

✅ R9 csv+axioms+markdown run 2 graded by code

S01 (K3).

JSON: {"source": "S01"}

✅ R9 csv+axioms+markdown run 3 graded by code

S01 (K3).

JSON: {"source": "S01"}

✅ R9 csv+axioms+markdown run 4 graded by code

S01 (K3).

JSON: {"source": "S01"}

✅ R9 csv+axioms+markdown run 5 graded by code

S01 (K3).

JSON: {"source": "S01"}