Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q44, run 2026-11-06-0855-sonnet

← back to the run

Which institutions are part of the Albury University System, at any level below it?

Tests: Local type: an affiliated institute (AFF) is not a unit of its parent.

Expected: institutions = ['I02', 'I03', 'I04', 'I05']

✅ R8 csv+axioms run 1 graded by code

I02, I03, I04, I05. I06 is an affiliated institute and does not count as a unit.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}

✅ R8 csv+axioms run 2 graded by code

I02, I03, I04, I05. I06 is only an affiliated institute and does not count as a unit.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}

✅ R8 csv+axioms run 3 graded by code

I02, I03, I04, I05. I06 is only an affiliated institute and is not counted as a unit.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}

✅ R8 csv+axioms run 4 graded by code

I02, I03, I04, I05. I06 has I02 as its record parent, but as an affiliated institute it is not a unit of I02.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}

✅ R8 csv+axioms run 5 graded by code

I02, I03, I04, I05. I06 is only an affiliated institute, so it is not a unit.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}

✅ R9 csv+axioms+markdown run 1 graded by code

I02, I03, I04, I05. I06 is an affiliated institute, so it is not part of the system.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}

✅ R9 csv+axioms+markdown run 2 graded by code

I02, I03, I04, I05. I06 is excluded because it is an affiliated institute.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}

✅ R9 csv+axioms+markdown run 3 graded by code

I02, I03, I04, I05. I06 is an affiliated institute and is not part of the system.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}

✅ R9 csv+axioms+markdown run 4 graded by code

I02, I03, I04, I05. I06 is an affiliated institute, so it is not a unit.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}

✅ R9 csv+axioms+markdown run 5 graded by code

I02, I03, I04, I05. I06 is an affiliated institute, so it is not a unit.

JSON: {"institutions": ["I02", "I03", "I04", "I05"]}