Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q2, run 2026-11-06-0855-sonnet

← back to the run

List every work, of any type, that Mara Ellison (A01) is an author of.

Tests: Identity: her works are spread over three author records, and A04's works (W08, W16) are not hers.

Expected: works = ['W01', 'W02', 'W03', 'W04', 'W05', 'W06', 'W07', 'W12', 'W19']

✅ R8 csv+axioms run 1 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. This takes the works of A01 plus A02 (W02, W07, W19) and A03 (W03), who share the ORCID.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R8 csv+axioms run 2 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 combined. A04 (W08, W16) has a different ORCID and is excluded.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R8 csv+axioms run 3 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 (the same person by ORCID); W08 and W16 belong to A04, a different person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R8 csv+axioms run 4 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03, who are the same person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R8 csv+axioms run 5 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03, which are the same person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R9 csv+axioms+markdown run 1 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works recorded against A01, A02 or A03. A04 has a different ORCID, so W08 and W16 are not included.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R9 csv+axioms+markdown run 2 graded by code

W01, W02, W03, W04, W05, W07, W06, W12, W19. A01 has W01, W04, W05, W06 and W12; A02 has W02, W07 and W19; A03 has W03.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R9 csv+axioms+markdown run 3 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19 (via A01, A02 and A03).

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R9 csv+axioms+markdown run 4 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01 plus those of A02 (W02, W07, W19) and A03 (W03), who are the same person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R9 csv+axioms+markdown run 5 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03, one person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}