Q31, run 2026-11-04-2010-sonnet

← back to the run

Which works cite at least one of the works that cite W01?

Tests: Two hops; W51 cites both first-hop works; W90 is a test record.

Expected: works = ['W50', 'W51', 'W52']

✅ R1 csv run 1 graded by code

W50, W51, W52. W90 also cites W41 but is a staging record.

JSON: {"works": ["W50", "W51", "W52"]}

❌ R1 csv run 2 graded by code

W50, W51, W52, W90. W90 is a staging record, so it is included here only because the question asks about all works.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

✅ R1 csv run 3 graded by code

W50, W51 and W52. W01 is cited by W40 and W41, and W90 (staging) is excluded.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R1 csv run 4 graded by code

W50, W51, W52. W40 and W41 cite W01. I excluded the staging record W90, which also cites W41.

JSON: {"works": ["W50", "W51", "W52"]}

❌ R1 csv run 5 graded by code

W50, W51, W52 and W90. The works citing W01 are W40 and W41. W90 is a staging record but is in the data.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

❌ R2 rdf run 1 graded by code

W50, W51, W52, W90. W01 is cited by W40 and W41.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

❌ R2 rdf run 2 graded by code

W50, W51, W52, W90. W01 is cited by W40 and W41. W40 is cited by W50 and W51, and W41 by W51, W52 and W90.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

❌ R2 rdf run 3 graded by code

W50, W51, W52, W90 (they cite W40 or W41, which cite W01).

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

❌ R2 rdf run 4 graded by code

W50, W51, W52, W90. W01 is cited by W40 and W41. W40 is cited by W50 and W51, and W41 by W51, W52 and W90.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

❌ R2 rdf run 5 graded by code

W50, W51, W52, W90. W01 is cited by W40 and W41, and these four cite one of those.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

✅ R6 rdf+ontology run 1 graded by code

W50, W51, W52. W90 is a staging record and is excluded.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R6 rdf+ontology run 2 graded by code

W50, W51, W52. W90 is excluded as a staging record.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R6 rdf+ontology run 3 graded by code

W50, W51, W52. W90 is excluded as a staging record.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R6 rdf+ontology run 4 graded by code

W50, W51, W52. W90 is a staging record and is excluded.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R6 rdf+ontology run 5 graded by code

W50, W51, W52. W90 is a staging record, so it is excluded.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R5 csv+ontology run 1 graded by code

W50, W51 and W52. W01 is cited by W40 and W41, and these are the research works that cite W40 or W41 (W90 is staging).

JSON: {"works": ["W50", "W51", "W52"]}

✅ R5 csv+ontology run 2 graded by code

W50, W51 and W52. W40 and W41 cite W01, and W90 is a staging record.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R5 csv+ontology run 3 graded by code

W50, W51, W52. W01 is cited by W40 and W41, and the works citing those are W50, W51 and W52. W90 is a staging record and is ignored.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R5 csv+ontology run 4 graded by code

W50, W51, W52. W01 is cited by W40 and W41, and W90 is staging so it is ignored.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R5 csv+ontology run 5 graded by code

W50, W51 and W52. W01 is cited by W40 and W41, and W90 is a staging record, so it is ignored.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R4 csv+markdown run 1 graded by code

W50, W51 and W52. Their cited works W40 and W41 are the works that cite W01.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R4 csv+markdown run 2 graded by code

W50, W51, W52. W90 is staging and excluded.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R4 csv+markdown run 3 graded by code

W50, W51, W52. W01 is cited by W40 and W41, and W90 is a staging record and is ignored.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R4 csv+markdown run 4 graded by code

W50, W51, W52. W01 is cited by W40 and W41, and these are the works citing them. W90 is a staging record and is excluded.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R4 csv+markdown run 5 graded by code

W50, W51, W52. W40 and W41 cite W01, and W90 is staging.

JSON: {"works": ["W50", "W51", "W52"]}

❌ R3 csv+padding run 1 graded by code

W50, W51, W52, W90. W01 is cited by W40 and W41, and these four cite one of them.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

❌ R3 csv+padding run 2 graded by code

W50, W51, W52, W90. W01 is cited by W40 and W41, and these four cite one or both of them.

JSON: {"works": ["W50", "W51", "W52", "W90"]}

Why: works: expected ['W50', 'W51', 'W52'], got ['W50', 'W51', 'W52', 'W90']

✅ R3 csv+padding run 3 graded by code

W50, W51, W52. W01 is cited by W40 and W41, and these three cite them.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R3 csv+padding run 4 graded by code

W50, W51, W52. W01 is cited by W40 and W41.

JSON: {"works": ["W50", "W51", "W52"]}

✅ R3 csv+padding run 5 graded by code

W50, W51 and W52. W01 is cited by W40 and W41, and these three cite those two.

JSON: {"works": ["W50", "W51", "W52"]}