Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q29, run 2026-11-04-2010-sonnet

← back to the run

Of the works citing "Drift-aware mapping with low-cost sonar" (either version), how many are self-citations at the level of authors, of the University of Albury (units included), and of countries?

Tests: Self-citation depends on the level; W12 is Mara Ellison under another record, W13 her co-author; W16 is the other M. Ellison.

Expected: author = 2 institution = 2 country = 4

✅ R1 csv run 1 graded by code

Author level 2 (W12, W13), University of Albury 2 (W12, W13), country 4 (W12, W13, W14, W15). W16 is by a different M. Ellison (A04), so it is not a self-citation.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R1 csv run 2 graded by code

Author 2, institution 2, country 4. Author-level: W12 and W13. Institution-level: W12 and W13. Country-level (GB): W12, W13, W14, W15. W16 is by A04, a different M. Ellison.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R1 csv run 3 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15). W16's author A04 is a different Ellison.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R1 csv run 4 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15, all GB). W16's M. Ellison (A04) is a different person.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R1 csv run 5 graded by code

Author 2, institution 2, country 4. The citing works are W12–W18.
- Author level: W12 (Ellison) and W13 (A07). W16's A04 is a different Ellison.
- Institution level: W12 and W13.
- Country level (GB): W12, W13, W14 and W15.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R2 rdf run 1 graded by code

Author 2, institution 2, country 4.
- **Author:** W12 and W13 share a person with the cited works (A01/A02/A03 and A07). W16's author A04 is a different person.
- **Institution:** W12 and W13 are I02 or its units.
- **Country:** W12, W13, W14 and W15 are all GB.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R2 rdf run 2 graded by code

Author 2, institution 2, country 4. The citing works are W12 to W18.
- Author-level: W12 (Mara) and W13 (Priya) share an author with the cited work.
- Institution-level: W12 (I02) and W13 (I03).
- Country-level: W12, W13, W14 (I05) and W15 (I13), all GB.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R2 rdf run 3 graded by code

Author level 2 (W12, W13), institution level 2 (W12, W13), country level 4 (W12, W13, W14, W15, all GB). W16's author A04 is a different "M. Ellison", so it is not an author self-citation.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R2 rdf run 4 graded by code

Author 2 (W12 and W13), institution 2 (W12 and W13), country 4 (W12, W13, W14 and W15, all GB). W16's author A04 is a different Ellison.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R2 rdf run 5 graded by code

Author 2, institution 2, country 4.
- Author level: W12 (A01) and W13 (A07) share an author with the cited versions; W16's A04 is a different person.
- Institution level: W12 (I02) and W13 (I03) are Albury; W14 is I05, which is not under I02.
- Country level: W12, W13, W14 and W15 are GB.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R6 rdf+ontology run 1 graded by code

Author 2 (W12, W13). Institution 2 (W12, W13). Country 4 (W12, W13, W14, W15).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R6 rdf+ontology run 2 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R6 rdf+ontology run 3 graded by code

Author 2, institution 2, country 4.
- Author level: W12 and W13 (Mara and Priya).
- Institution level: the same two.
- Country level: W12, W13, W14 and W15 (all GB).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R6 rdf+ontology run 4 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R6 rdf+ontology run 5 graded by code

Author 2 (W12, W13). Institution 2 (W12, W13). Country 4 (W12, W13, W14, W15, all GB).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R5 csv+ontology run 1 graded by code

Author 2, institution 2, country 4. At author and institution level the self-citing works are W12 and W13. At country level (GB) W14 and W15 are added.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R5 csv+ontology run 2 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15). W16 is by A04, a different person from Mara.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R5 csv+ontology run 3 graded by code

Author level: 2 (W12 and W13). University of Albury level: 2 (W12 and W13, the latter via I03). Country level: 4 (W12, W13, W14, W15, all GB, which gives 4).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R5 csv+ontology run 4 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15). W16's author is A04, a different person. W14 is credited to I05, which is not under I02, but it is a GB institution. W15 is credited to I13, also GB.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R5 csv+ontology run 5 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R4 csv+markdown run 1 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15, all GB).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R4 csv+markdown run 2 graded by code

Author 2 (W12, W13). Institution 2 (W12, W13). Country 4 (W12, W13, W14, W15, all GB).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R4 csv+markdown run 3 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15). W16 is by A04, a different person.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R4 csv+markdown run 4 graded by code

Author 2, institution 2, country 4.
- Author level: W12 and W13.
- Institution level (I02): W12 and W13.
- Country level (GB): W12, W13, W14 and W15.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R4 csv+markdown run 5 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15). The citing works are W12–W18.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R3 csv+padding run 1 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15).

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R3 csv+padding run 2 graded by code

Author level 2 (W12, W13), institution level 2 (W12, W13), country level 4 (W12, W13, W14, W15). W16 is by the other M. Ellison (A04), so it is not a self-citation.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R3 csv+padding run 3 graded by code

Author 2, institution 2, country 4. Authors: W12 and W13. Institutions: W12 (I02) and W13 (I03). Countries: W12, W13, W14 and W15 are all GB.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R3 csv+padding run 4 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15). W16's author A04 is a different Ellison. W14 is I05, which is outside the University of Albury but still in GB.

JSON: {"author": 2, "institution": 2, "country": 4}

✅ R3 csv+padding run 5 graded by code

Author 2 (W12, W13), institution 2 (W12, W13), country 4 (W12, W13, W14, W15). W16's A04 is a different person from Mara Ellison.

JSON: {"author": 2, "institution": 2, "country": 4}