Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q2, run 2026-11-04-2010-sonnet

← back to the run

List every work, of any type, that Mara Ellison (A01) is an author of.

Tests: Identity: her works are spread over three author records, and A04's works (W08, W16) are not hers.

Expected: works = ['W01', 'W02', 'W03', 'W04', 'W05', 'W06', 'W07', 'W12', 'W19']

✅ R1 csv run 1 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. I read this as the person behind A01, A02 and A03. A01's own record alone covers only W01, W04, W05, W06 and W12.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R1 csv run 2 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works under A01 and its aliases A02 and A03.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R1 csv run 3 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 (same person); W08 and W16 belong to A04, a different person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R1 csv run 4 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works by A01, A02 and A03, which are the same person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R1 csv run 5 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These come from A01, A02 and A03. W08 and W16 belong to A04, a different person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R2 rdf run 1 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These come from A01 (W01, W04, W05, W06, W12), A02 (W02, W07, W19) and A03 (W03), all the same person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R2 rdf run 2 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. Mara is A01, A02 and A03 combined: A01 wrote W01, W04, W05, W06 and W12, A02 wrote W02, W07 and W19, and A03 wrote W03.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R2 rdf run 3 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. A01 authored W01, W04, W05, W06 and W12. A02 authored W02, W07 and W19, and A03 authored W03.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R2 rdf run 4 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. A01 alone has W01, W04, W05, W06 and W12. The same person under A02 adds W02, W07 and W19, and under A03 adds W03.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R2 rdf run 5 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. This takes Mara Ellison as the person behind A01, A02 and A03 (shared ORCID), not A01's records alone, which would give W01, W04, W05, W06, W12.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R6 rdf+ontology run 1 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 together.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R6 rdf+ontology run 2 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These come from A01 (W01, W04, W05, W06, W12), A02 (W02, W07, W19) and A03 (W03).

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R6 rdf+ontology run 3 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 (A01: W01, W04, W05, W06, W12; A02: W02, W07, W19; A03: W03).

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R6 rdf+ontology run 4 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03, which are one person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R6 rdf+ontology run 5 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the authorships of A01 (W01, W04, W05, W06, W12), A02 (W02, W07, W19) and A03 (W03).

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R5 csv+ontology run 1 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works under A01, A02 and A03; W08 is by A04, a different person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R5 csv+ontology run 2 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works recorded against A01, A02 or A03, all the same person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R5 csv+ontology run 3 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works recorded against A01, A02 and A03. W08 and W16 belong to A04, a different person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R5 csv+ontology run 4 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. A01's own records give W01, W04, W05, W06 and W12. A02 adds W02, W07 and W19, and A03 adds W03.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R5 csv+ontology run 5 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works recorded against A01 (W01, W04, W05, W06, W12), A02 (W02, W07, W19) and A03 (W03).

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R4 csv+markdown run 1 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 together, since one ORCID makes them the same person. A04 has a different ORCID and is a different person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R4 csv+markdown run 2 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03, all one person. A04 (a different ORCID) is excluded.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R4 csv+markdown run 3 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. A01, A02 and A03 are the same person, so all their records count, whatever the work type.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R4 csv+markdown run 4 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 together.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R4 csv+markdown run 5 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 together.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R3 csv+padding run 1 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R3 csv+padding run 2 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 combined: A01 has W01, W04, W05, W06, W12; A02 has W02, W07, W19; A03 has W03.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R3 csv+padding run 3 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. A01 authored W01, W04, W05, W06 and W12, A02 authored W02, W07 and W19, and A03 authored W03.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R3 csv+padding run 4 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. These are the works of A01, A02 and A03 (the same person). W08 and W16 belong to A04, a different person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}

✅ R3 csv+padding run 5 graded by code

W01, W02, W03, W04, W05, W06, W07, W12, W19. A01, A02 and A03 are one person; W08 and W16 belong to A04, a different person.

JSON: {"works": ["W01", "W02", "W03", "W04", "W05", "W06", "W07", "W12", "W19"]}