Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q9, run 2026-11-04-1825-sonnet

← back to the run

As of 1 October 2026, how many distinct issuers hold an issuer rating?

Tests: Everything at once: identity, withdrawal, instrument versus issuer, ceiling versus rating.

Expected: count = 12

❌ R1 csv run 1 graded by code

13. Counting E03 and E02 once, the rated issuers are E01, E02, E05, E06, E08, E09, E10, E12, E13, E14, E15, E16 and CV. E07 was withdrawn (A19), and the CC rows and E11's CO row are not issuer ratings.

JSON: {"count": 13}

Why: count: expected 12, got 13

❌ R1 csv run 2 graded by code

13. These are E01, E02 (E03 is the same entity), E05, E06, E08, E09, E10, E12, E13, E14, E15, E16 and CV. E07 is withdrawn (A19), VD has only a Country Ceiling, and E11 has only a CO record.

JSON: {"count": 13}

Why: count: expected 12, got 13

❌ R1 csv run 3 graded by code

13. The issuers are E01, E02 (E03 is the same LEI), E05, E06, E08, E09, E10, E12, E13, E14, E15, E16 and CV. E07 is withdrawn, E04 and E11 have no LTIDR, and VD has only a country ceiling.

JSON: {"count": 13}

Why: count: expected 12, got 13

❌ R1 csv run 4 graded by code

13. These are E01, E02 (E03 is a duplicate), E05, E06, E08, E09, E10, E12, E13, E14, E15, E16 and CV. E07 was withdrawn (A19). VD has only a country ceiling (A12) and E11 only a CO record (A46), neither of which I counted as an issuer rating.

JSON: {"count": 13}

Why: count: expected 12, got 13

❌ R1 csv run 5 graded by code

13. The issuers are E01, E02 (E03 is the same entity), E05, E06, E08, E09, E10, E12, E13, E14, E15, E16 and CV. E07 was withdrawn (A19). VD has only a country ceiling (CC), and E11 has only a CO-type rating (A46).

JSON: {"count": 13}

Why: count: expected 12, got 13

✅ R5 csv+ontology run 1 graded by code

12. These are E01, E02 (with E03), E05, E06, E08, E09, E10, E12, E13, E14, CV and E15. E07 is withdrawn, E11 has only a credit opinion, and VD has only a ceiling.

JSON: {"count": 12}

✅ R5 csv+ontology run 2 graded by code

12. These are E01, E02 (with E03), E05, E06, E08 (RD), E09, E10 (with E17), E12, E13, E14, CV and E15. I excluded E07 (withdrawn), E04, E11 (credit opinion only), VD (ceiling only) and E16 (QA).

JSON: {"count": 12}

✅ R5 csv+ontology run 3 graded by code

12. These are E01, E02 (with E03), E05, E06, E08, E09, E10, E12, E13, E14, CV and E15. E07 was withdrawn (A19), E04 and E11 have no issuer rating (E11 has only a credit opinion, A46), and VD has only a ceiling.

JSON: {"count": 12}

✅ R5 csv+ontology run 4 graded by code

12. These are E01, E02 (same entity as E03), E05, E06, E08, E09, E10 (same entity as E17), E12, E13, E14, CV and E15. E07 is withdrawn, E04 and E11 have no issuer rating (E11 has only a credit opinion), VD has only a ceiling, and E16 is QA.

JSON: {"count": 12}

✅ R5 csv+ontology run 5 graded by code

12. The issuers with a LTIDR on 2026-10-01 are E01, E02 (E03 is the same entity), E05, E06, E08, E09, E10, E12, E13, E14, CV and E15. E07 was withdrawn, E11 has only a credit opinion, VD has only a ceiling, and E16 is QA.

JSON: {"count": 12}

✅ R8 csv+axioms run 1 graded by code

12. These are E01, E02 (same entity as E03), E05, E06, E08, E09, E10, E12, E13, E14, CV and E15. VD has only a ceiling (CC), E04 and E11 have no issuer ratings, E07's was withdrawn, and E16 is a test record.

JSON: {"count": 12}

✅ R8 csv+axioms run 2 graded by code

12. These are E01, E02 (E03 is the same entity), E05, E06, E08, E09, E10, E12, E13, E14, E15 and CV. E07 is withdrawn (A19), E11 has only a credit opinion (A46), E16 is a test record, and VD has only a ceiling (A12).

JSON: {"count": 12}

✅ R8 csv+axioms run 3 graded by code

12. These are E01, E02 (E03 is the same entity), E05, E06, E08, E09, E10, E12, E13, E14, CV and E15. E07's rating was withdrawn (A19), E11 has only a credit opinion, VD has only a ceiling, and E16 is a test record.

JSON: {"count": 12}

✅ R8 csv+axioms run 4 graded by code

12. These are E01, E02 (merged with E03), E05, E06, E08, E09, E10, E12, E13, E14, CV and E15.

JSON: {"count": 12}

✅ R8 csv+axioms run 5 graded by code

12. These are E01, E02 (the same entity as E03), E05, E06, E08, CV, E09, E10, E12, E13, E14 and E15. E07 is withdrawn (A19), E16 is a test record, and E04, E11 and VD have no issuer rating in force.

JSON: {"count": 12}

✅ R9 csv+axioms+markdown run 1 graded by code

12. The issuers are E01, E02 (with E03), E05, E06, E08, E09, E10 (with E17), E12, E13, E14, CV and E15. E07 was withdrawn, E16 is QA, and E04, E11 and VD have no issuer rating.

JSON: {"count": 12}

✅ R9 csv+axioms+markdown run 2 graded by code

12. These are E01, E02 (with E03), E05, E06, E08, E09, E10, E12, E13, E14, CV and E15. E07 is withdrawn, E04 and E11 have no issuer rating (E11 has only a credit opinion), VD has only a ceiling, and E16 is QA.

JSON: {"count": 12}

✅ R9 csv+axioms+markdown run 3 graded by code

12. These are E01, E02 (with E03), E05, E06, E08, E09, E10 (with E17), E12, E13, E14, CV and E15. E07 is withdrawn, VD has only a ceiling, E11 has only a credit opinion, E04 has no rating, and E16 is QA.

JSON: {"count": 12}

✅ R9 csv+axioms+markdown run 4 graded by code

12. These are E01, E02 (with E03), E05, E06, E08, E09, E10 (with E17), E12, E13, E14, CV and E15. E04, E11 and VD have no issuer rating, E07 is withdrawn and E16 is QA.

JSON: {"count": 12}

✅ R9 csv+axioms+markdown run 5 graded by code

12. These are E01, E02 (with E03), E05, E06, E08, E09, E10 (with E17), E12, E13, E14, CV and E15. E07 is withdrawn, E16 is QA, and E04, E11 and VD have no issuer rating.

JSON: {"count": 12}