Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q22, run 2026-11-04-1712-sonnet

← back to the run

Which entities have ever been in default?

Tests: Default semantics: RD is a default; WD is not.

Expected: entities = ['E08']

✅ R1 csv run 1 graded by code

E08, which was rated RD (A22).

JSON: {"entities": ["E08"]}

✅ R1 csv run 2 graded by code

E08. It was rated RD in A22.

JSON: {"entities": ["E08"]}

✅ R1 csv run 3 graded by code

E08, which was rated RD in A22.

JSON: {"entities": ["E08"]}

✅ R1 csv run 4 graded by code

E08, which was rated RD in A22.

JSON: {"entities": ["E08"]}

✅ R1 csv run 5 graded by code

E08, which went to RD (A22).

JSON: {"entities": ["E08"]}

✅ R10 executed run 1 graded by code

E08 (RD, A22).

JSON: {"entities": ["E08"]}

✅ R10 executed run 2 graded by code

E08 (RD, A22).

JSON: {"entities": ["E08"]}

✅ R10 executed run 3 graded by code

E08 (RD, A22).

JSON: {"entities": ["E08"]}

✅ R10 executed run 4 graded by code

E08, which reached RD (A22).

JSON: {"entities": ["E08"]}

✅ R10 executed run 5 graded by code

E08 (RD, A22).

JSON: {"entities": ["E08"]}