Semantics ladder

How much does an LLM gain from being told what data means?

The experiment

Goals
Method

Credit ratings

Research analytics

The results

All data is invented: the companies, LEIs, ratings, works, people and institutions. Nothing here is any rating agency's or bibliographic database's data or definitions.

Last updated 6 November 2026.

Q19, run 2026-11-04-1712-sonnet

← back to the run

Which issuers had a Negative outlook on 1 October 2026?

Tests: Outlooks: set by affirmations, without a rating change.

Expected: issuers = ['E01', 'E09']

✅ R1 csv run 1 graded by code

E01 (A03) and E09 (A25).

JSON: {"issuers": ["E01", "E09"]}

✅ R1 csv run 2 graded by code

E01, E09. A03 and A25 give them NEG outlooks, and E06 has no outlook after A17.

JSON: {"issuers": ["E01", "E09"]}

✅ R1 csv run 3 graded by code

E01 (A03) and E09 (A25).

JSON: {"issuers": ["E01", "E09"]}

✅ R1 csv run 4 graded by code

E01 (A03) and E09 (A25).

JSON: {"issuers": ["E01", "E09"]}

✅ R1 csv run 5 graded by code

E01 (A03) and E09 (A25).

JSON: {"issuers": ["E01", "E09"]}

✅ R10 executed run 1 graded by code

E01 (A03) and E09 (A25). E06's outlook is blank because it is on Watch.

JSON: {"issuers": ["E01", "E09"]}

✅ R10 executed run 2 graded by code

E01 (A03) and E09 (A25).

JSON: {"issuers": ["E01", "E09"]}

✅ R10 executed run 3 graded by code

E01 and E09 (A03, A25). E06 has no outlook while on watch.

JSON: {"issuers": ["E01", "E09"]}

✅ R10 executed run 4 graded by code

E01 (A03) and E09 (A25).

JSON: {"issuers": ["E01", "E09"]}

✅ R10 executed run 5 graded by code

E01 (A03) and E09 (A25).

JSON: {"issuers": ["E01", "E09"]}