Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q17, run 2026-11-04-2053-sonnet

← back to the run

Which of Mara Ellison's works are not research output because of their document type?

Tests: Document types: an editorial and an erratum, under two author records.

Expected: works = ['W04', 'W05']

✅ R1 csv run 1 graded by code

W04 (editorial) and W05 (erratum).

JSON: {"works": ["W04", "W05"]}

✅ R1 csv run 2 graded by code

W04 (EDT) and W05 (ERR).

JSON: {"works": ["W04", "W05"]}

✅ R1 csv run 3 graded by code

W04 (editorial) and W05 (erratum).

JSON: {"works": ["W04", "W05"]}

✅ R1 csv run 4 graded by code

W04 (editorial) and W05 (erratum).

JSON: {"works": ["W04", "W05"]}

✅ R1 csv run 5 graded by code

W04 (editorial) and W05 (erratum).

JSON: {"works": ["W04", "W05"]}

✅ R10 executed run 1 graded by code

W04 (editorial) and W05 (erratum). W07 is excluded because its source is delisted, not because of its type.

JSON: {"works": ["W04", "W05"]}

✅ R10 executed run 2 graded by code

W04 (EDT) and W05 (ERR). W07 is excluded because its source is delisted, not because of its type.

JSON: {"works": ["W04", "W05"]}

✅ R10 executed run 3 graded by code

W04 (editorial) and W05 (erratum). W07 is excluded for a different reason: its source is delisted.

JSON: {"works": ["W04", "W05"]}

✅ R10 executed run 4 graded by code

W04 (editorial) and W05 (erratum). W07 is excluded because its source is delisted, not because of its type.

JSON: {"works": ["W04", "W05"]}

✅ R10 executed run 5 graded by code

W04 (EDT) and W05 (ERR).

JSON: {"works": ["W04", "W05"]}