Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q30, run 2026-11-05-1227-sonnet

← back to the run

Which entities had a foreign-currency issuer rating exactly at their country's ceiling on 1 October 2026?

Tests: Ceilings: which ratings the cap binds.

Expected: entities = ['E05', 'E15']

✅ R1 csv run 1 graded by code

E05 and E15. E05's foreign-currency BB equals VD's ceiling of BB, and E15's foreign-currency A equals CV's ceiling of A (A37).

JSON: {"entities": ["E05", "E15"]}

✅ R1 csv run 2 graded by code

E05 and E15. E05's foreign-currency rating is BB (A14) against a VD ceiling of BB, and E15's is A (A41) against a CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R1 csv run 3 graded by code

E05 and E15. E05's foreign-currency rating of BB equals the VD ceiling of BB, and E15's A equals the CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R1 csv run 4 graded by code

E05 (FC BB equals the VD ceiling of BB) and E15 (FC A equals the CV ceiling of A).

JSON: {"entities": ["E05", "E15"]}

✅ R1 csv run 5 graded by code

E05 and E15. E05's foreign-currency rating of BB equals VD's ceiling of BB, and E15's A equals CV's ceiling of A (A37).

JSON: {"entities": ["E05", "E15"]}

✅ R2 rdf run 1 graded by code

E05 (FC BB = VD ceiling BB) and E15 (FC A = CV ceiling A) (A12, A14, A37, A41).

JSON: {"entities": ["E05", "E15"]}

✅ R2 rdf run 2 graded by code

E05 and E15. E05's FC rating BB equals VD's ceiling BB (A14, A12). E15's FC rating A equals CV's ceiling A (A41, A37).

JSON: {"entities": ["E05", "E15"]}

✅ R2 rdf run 3 graded by code

E05 and E15. The ceilings are VD = BB and CV = A, and E05 FC is BB (A14) and E15 FC is A (A41). CV's own FC rating is BBB+, so it is not at the ceiling.

JSON: {"entities": ["E05", "E15"]}

✅ R2 rdf run 4 graded by code

E05 (FC BB, A14) and E15 (FC A, A41). They match the BB ceiling for VD (A12) and the A ceiling for CV (A37).

JSON: {"entities": ["E05", "E15"]}

✅ R2 rdf run 5 graded by code

E05 and E15. E05's foreign-currency rating is BB against VD's ceiling of BB (A14, A12). E15's is A against CV's ceiling of A (A41, A37).

JSON: {"entities": ["E05", "E15"]}

✅ R6 rdf+ontology run 1 graded by code

E05 and E15. E05's FC rating of BB (A14) equals Valdoria's ceiling of BB (A12). E15's FC rating of A (A41) equals Corvenia's ceiling of A (A37).

JSON: {"entities": ["E05", "E15"]}

✅ R6 rdf+ontology run 2 graded by code

E05 and E15. E05's FC rating is BB, equal to VD's ceiling of BB (A14, A12), and E15's FC rating is A, equal to CV's ceiling of A (A41, A37).

JSON: {"entities": ["E05", "E15"]}

✅ R6 rdf+ontology run 3 graded by code

E05 and E15. E05's FC rating is BB, equal to the VD ceiling of BB (A14, A12). E15's FC rating is A, equal to the CV ceiling of A (A41, A37).

JSON: {"entities": ["E05", "E15"]}

✅ R6 rdf+ontology run 4 graded by code

E05 and E15. E05's FC rating of BB equals VD's ceiling (A12), and E15's FC rating of A equals CV's ceiling (A37).

JSON: {"entities": ["E05", "E15"]}

✅ R6 rdf+ontology run 5 graded by code

E05 and E15. E05's foreign-currency rating of BB equals the VD ceiling of BB (A14, A12), and E15's A equals the CV ceiling of A (A41, A37).

JSON: {"entities": ["E05", "E15"]}

✅ R5 csv+ontology run 1 graded by code

E05 and E15. E05's FC rating is BB, equal to the VD ceiling BB (A12, A14), and E15's FC rating is A, equal to the CV ceiling A (A37, A41).

JSON: {"entities": ["E05", "E15"]}

✅ R5 csv+ontology run 2 graded by code

E05 and E15. E05's FC rating of BB equals VD's ceiling of BB, and E15's FC rating of A equals CV's ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R5 csv+ontology run 3 graded by code

E05 and E15. E05's FC rating of BB equals VD's ceiling of BB, and E15's FC rating of A equals CV's ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R5 csv+ontology run 4 graded by code

E05 and E15. E05's FC rating of BB equals VD's ceiling of BB, and E15's FC rating of A equals CV's ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R5 csv+ontology run 5 graded by code

E05 and E15. E05's FC rating of BB (A14) equals Valdoria's ceiling of BB, and E15's FC rating of A (A41) equals Corvenia's ceiling of A (A37). The sovereign CV's FC rating of BBB+ is below its ceiling.

JSON: {"entities": ["E05", "E15"]}

✅ R4 csv+markdown run 1 graded by code

E05 (FC BB equals the VD ceiling BB) and E15 (FC A equals the CV ceiling A).

JSON: {"entities": ["E05", "E15"]}

✅ R4 csv+markdown run 2 graded by code

E05 (FC BB, VD ceiling BB) and E15 (FC A, CV ceiling A).

JSON: {"entities": ["E05", "E15"]}

✅ R4 csv+markdown run 3 graded by code

E05 and E15. E05's FC rating is BB (A14) against the VD ceiling of BB (A12), and E15's is A (A41) against the CV ceiling of A (A37).

JSON: {"entities": ["E05", "E15"]}

✅ R4 csv+markdown run 4 graded by code

E05 and E15. E05's foreign-currency rating of BB equals the VD ceiling of BB, and E15's A equals the CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R4 csv+markdown run 5 graded by code

E05 (FC BB, equal to the VD ceiling of BB) and E15 (FC A, equal to the CV ceiling of A).

JSON: {"entities": ["E05", "E15"]}

✅ R3 csv+padding run 1 graded by code

E05 and E15. E05's FC rating is BB, equal to VD's ceiling of BB, and E15's FC rating is A, equal to CV's ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R3 csv+padding run 2 graded by code

E05 (BB, matching the VD ceiling of BB) and E15 (A, matching the CV ceiling of A, A37). Ceilings for other countries are not in the data.

JSON: {"entities": ["E05", "E15"]}

✅ R3 csv+padding run 3 graded by code

E05 (BB, A14, against VD's ceiling of BB) and E15 (A, A41, against CV's ceiling of A).

JSON: {"entities": ["E05", "E15"]}

✅ R3 csv+padding run 4 graded by code

E05 and E15. E05's FC rating of BB equals the Valdoria ceiling of BB (A14, A12). E15's FC rating of A equals the Corvenia ceiling of A (A41, A37).

JSON: {"entities": ["E05", "E15"]}

✅ R3 csv+padding run 5 graded by code

E05 (FC BB, equal to VD's ceiling of BB) and E15 (FC A, equal to CV's ceiling of A).

JSON: {"entities": ["E05", "E15"]}

✅ R10 executed run 1 graded by code

E05 and E15. E05's foreign-currency rating is BB, equal to the VD ceiling (A14). E15's is A, equal to the CV ceiling (A41).

JSON: {"entities": ["E05", "E15"]}

✅ R10 executed run 2 graded by code

E05 and E15. E05's foreign-currency rating of BB equals the VD ceiling of BB, and E15's A equals the CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R10 executed run 3 graded by code

Valdoria Telecom (E05) and Corvenia Power (E15). E05's FC rating is BB against a BB ceiling, and E15's is A against an A ceiling.

JSON: {"entities": ["E05", "E15"]}

✅ R10 executed run 4 graded by code

E05 and E15. E05's FC rating is BB, equal to the VD ceiling of BB, and E15's is A, equal to the CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R10 executed run 5 graded by code

E05 and E15. E05's FC rating of BB equals VD's BB ceiling, and E15's FC rating of A equals CV's A ceiling.

JSON: {"entities": ["E05", "E15"]}

✅ R7 csv+neutral run 1 graded by code

E05 and E15. E05's FC rating is BB, equal to the VD ceiling (A12). E15's FC rating is A, equal to the CV ceiling (A37).

JSON: {"entities": ["E05", "E15"]}

✅ R7 csv+neutral run 2 graded by code

E05 has FC BB, equal to VD's ceiling of BB. E15 has FC A, equal to CV's ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R7 csv+neutral run 3 graded by code

E05 and E15. E05's foreign-currency rating of BB (A14) equals the VD ceiling of BB, and E15's A (A41) equals the CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R7 csv+neutral run 4 graded by code

E05 and E15. VD's ceiling is BB and E05's FC rating is BB. CV's ceiling is A and E15's FC rating is A.

JSON: {"entities": ["E05", "E15"]}

✅ R7 csv+neutral run 5 graded by code

E05 and E15. E05's FC rating is BB, equal to Valdoria's ceiling of BB, and E15's FC rating is A, equal to Corvenia's ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R8 csv+axioms run 1 graded by code

E05 and E15. E05's FC rating is BB, equal to the VD ceiling of BB (A12, A14). E15's is A, equal to the CV ceiling of A (A37, A41).

JSON: {"entities": ["E05", "E15"]}

✅ R8 csv+axioms run 2 graded by code

E05 and E15. E05's FC rating of BB equals the VD ceiling of BB, and E15's FC rating of A equals the CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R8 csv+axioms run 3 graded by code

E05 and E15. E05 is BB against the VD ceiling of BB, and E15 is A against the CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R8 csv+axioms run 4 graded by code

E05 (FC BB, VD ceiling BB) and E15 (FC A, CV ceiling A).

JSON: {"entities": ["E05", "E15"]}

✅ R8 csv+axioms run 5 graded by code

E05 and E15. E05's FC rating is BB, equal to the VD ceiling of BB, and E15's FC rating is A, equal to the CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R9 csv+axioms+markdown run 1 graded by code

E05 and E15. E05's FC rating is BB against a VD ceiling of BB (A14, A12), and E15's is A against a CV ceiling of A (A41, A37).

JSON: {"entities": ["E05", "E15"]}

✅ R9 csv+axioms+markdown run 2 graded by code

E05 and E15. E05's FC rating is BB, equal to the VD ceiling of BB, and E15's FC rating is A, equal to the CV ceiling of A.

JSON: {"entities": ["E05", "E15"]}

✅ R9 csv+axioms+markdown run 3 graded by code

E05 (FC BB against a VD ceiling of BB) and E15 (FC A against a CV ceiling of A).

JSON: {"entities": ["E05", "E15"]}

✅ R9 csv+axioms+markdown run 4 graded by code

E05 (FC BB, ceiling BB) and E15 (FC A, ceiling A).

JSON: {"entities": ["E05", "E15"]}

✅ R9 csv+axioms+markdown run 5 graded by code

E05 and E15. E05's FC rating is BB, equal to Valdoria's ceiling of BB (A12, A14). E15's FC rating is A, equal to Corvenia's ceiling of A (A37, A41).

JSON: {"entities": ["E05", "E15"]}