Q31, run 2026-11-04-2053-sonnet

← back to the run

Which works cite at least one of the works that cite W01?

Tests: Two hops; W51 cites both first-hop works; W90 is a test record.

Expected: works = ['W50', 'W51', 'W52']

✅ R1 csv run 1 graded by code

W50, W51, W52. W01 is cited by W40 and W41. The staging record W90 also cites W41 and is excluded.

JSON: {"works": ["W50", "W51", "W52"]}

❌ R1 csv run 2 graded by code

W50, W51, W52, W90.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

❌ R1 csv run 3 graded by code

W50, W51, W52, W90. W40 and W41 cite W01. W90 is a staging record, but the question is about works, not research output.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

✅ R1 csv run 4 graded by code

W50, W51, W52. W90 is also a citer but is a staging record.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R1 csv run 5 graded by code

W50, W51, W52. W01 is cited by W40 and W41, and these three cite those.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R10 executed run 1 graded by code

W50, W51, W52. W40 and W41 cite W01, and these three cite them.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R10 executed run 2 graded by code

W50, W51, W52. W01 is cited by W40 and W41, and those two are in turn cited by W50, W51 and W52.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R10 executed run 3 graded by code

W50, W51, W52. W01 is cited by W40 and W41, and those are in turn cited by W50, W51 and W52.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R10 executed run 4 graded by code

W50, W51, W52.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R10 executed run 5 graded by code

W50, W51, W52. W01 is cited by W40 and W41, and these three works cite W40 or W41.

JSON: {"works": ["W50", "W51", "W52"]}