Q31, run 2026-11-06-0206-haiku

← back to the run

Which works cite at least one of the works that cite W01?

Tests: Two hops; W51 cites both first-hop works; W90 is a test record.

Expected: works = ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895']

❌ R1 csv run 1 graded by code

W50, W51, W52, W90, W337, W338, W396, W546, W682, W801, W844, W895, W922, W1042 (14 works). These cite W40, W41, W229, W602, W657, W660, W844 or W895, all of which cite W01.

JSON: {"works": ["W50", "W51", "W52", "W90", "W337", "W338", "W396", "W546", "W682", "W801", "W844", "W895", "W922", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W90', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W844', 'W895', 'W922', 'W1042']

❌ R1 csv run 2 graded by code

14 works: W50, W51, W52, W90, W337, W338, W396, W546, W682, W801, W895, W922, W1039, W1042. Each cites one of the eight works that cite W01 (W40, W41, W229, W602, W657, W660, W844, W895).

JSON: {"works": ["W50", "W51", "W52", "W90", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W922", "W1039", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W90', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W895', 'W922', 'W1039', 'W1042']

❌ R1 csv run 3 graded by code

14 works cite at least one citer of W01: W50, W51, W52, W90, W337, W338, W396, W546, W682, W801, W844, W895, W922, W1042.

JSON: {"works": ["W50", "W51", "W52", "W90", "W337", "W338", "W396", "W546", "W682", "W801", "W844", "W895", "W922", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W90', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W844', 'W895', 'W922', 'W1042']

❌ R1 csv run 4 graded by code

(no answer found)

Why: no structured answer

❌ R1 csv run 5 graded by code

(no answer found)

JSON: {"works": ["W50", "W51", "W52", "W90", "W337", "W338", "W396", "W546", "W682", "W801", "W844", "W895", "W922", "W1039", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W90', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W844', 'W895', 'W922', 'W1039', 'W1042']

❌ R3 csv+padding run 1 graded by code

14 works cite at least one of W40, W41, W229, W602, W657, W660, W844 or W895: W50, W51, W52, W90, W337, W338, W396, W546, W682, W801, W844, W895, W922, W1042.

JSON: {"works": ["W50", "W51", "W52", "W90", "W337", "W338", "W396", "W546", "W682", "W801", "W844", "W895", "W922", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W90', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W844', 'W895', 'W922', 'W1042']

❌ R3 csv+padding run 2 graded by code

14 works: W50, W51, W52, W90, W337, W338, W396, W546, W682, W801, W844, W895, W922, W1042.

JSON: {"works": ["W50", "W51", "W52", "W90", "W337", "W338", "W396", "W546", "W682", "W801", "W844", "W895", "W922", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W90', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W844', 'W895', 'W922', 'W1042']

❌ R3 csv+padding run 3 graded by code

14 works cite at least one of the works that cite W01: W50, W51, W52, W90, W337, W338, W396, W546, W682, W801, W844, W895, W922, W1042.

JSON: {"works": ["W50", "W51", "W52", "W90", "W337", "W338", "W396", "W546", "W682", "W801", "W844", "W895", "W922", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W90', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W844', 'W895', 'W922', 'W1042']

❌ R3 csv+padding run 4 graded by code

W50, W51, W52, W90, W337, W338, W396, W546, W682, W801, W844, W895, W922, W1039, W1042.

JSON: {"works": ["W50", "W51", "W52", "W90", "W337", "W338", "W396", "W546", "W682", "W801", "W844", "W895", "W922", "W1039", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W90', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W844', 'W895', 'W922', 'W1039', 'W1042']

❌ R3 csv+padding run 5 graded by code

14 works cite at least one citer of W01: W50, W51, W52, W90, W337, W338, W396, W546, W682, W801, W844, W895, W922, W1042.

JSON: {"works": ["W50", "W51", "W52", "W90", "W337", "W338", "W396", "W546", "W682", "W801", "W844", "W895", "W922", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W90', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W844', 'W895', 'W922', 'W1042']

❌ R4 csv+markdown run 1 graded by code

(no answer found)

Why: no structured answer

❌ R4 csv+markdown run 2 graded by code

(no answer found)

Why: no structured answer

✅ R4 csv+markdown run 3 graded by code

11 works. W844 is STG and excluded; W922 is retracted and excluded.

JSON: {"works": ["W50", "W51", "W52", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W1042"]}

❌ R4 csv+markdown run 4 graded by code

(no answer found)

Why: no structured answer

✅ R4 csv+markdown run 5 graded by code

11 works: W50, W51, W52, W337, W338, W396, W546, W682, W801, W895, W1042. W844 and W90 are STG and excluded; W922 is retracted.

JSON: {"works": ["W50", "W51", "W52", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W1042"]}

❌ R5 csv+ontology run 1 graded by code

(no answer found)

Why: no structured answer

✅ R5 csv+ontology run 2 graded by code

W50, W51, W52, W337, W338, W396, W546, W682, W801, W895, W1042. STG W90 and W844 and retracted W922 are excluded.

JSON: {"works": ["W50", "W51", "W52", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W1042"]}

❌ R5 csv+ontology run 3 graded by code

W337, W682, W801, W1042 (W844 excluded as STG).

JSON: {"works": ["W337", "W682", "W801", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W337', 'W682', 'W801', 'W1042']

❌ R5 csv+ontology run 4 graded by code

(no answer found)

JSON: {"works": null}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got None

❌ R5 csv+ontology run 5 graded by code

12 works. They cite W40, W41, W229, W602, W660 or W895, which are themselves citers of W01.

JSON: {"works": ["W50", "W51", "W52", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W1039", "W1042"]}

Why: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W895', 'W1039', 'W1042']

✅ R10 executed run 1 graded by code

W50, W51, W52, W337, W338, W396, W546, W682, W801, W895 and W1042. Each cites one of W01's citers: W40, W41, W229, W602, W657, W660 or W895.

JSON: {"works": ["W50", "W51", "W52", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W1042"]}

✅ R10 executed run 2 graded by code

11 works: W50, W51, W52, W337, W338, W396, W546, W682, W801, W895, W1042. Each cites one of the citers of W01 (W40, W41, W229, W602, W657, W660, W895).

JSON: {"works": ["W50", "W51", "W52", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W1042"]}

✅ R10 executed run 3 graded by code

W50, W51, W52, W337, W338, W396, W546, W682, W801, W895, W1042. These cite W40, W41, W229, W602, W660 or W895, which are the works that cite W01.

JSON: {"works": ["W50", "W51", "W52", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W1042"]}

✅ R10 executed run 4 graded by code

W50, W51, W52, W337, W338, W396, W546, W682, W801, W895, W1042.
These cite W40, W41, W229, W602, W660 or W895, all of which cite W01.

JSON: {"works": ["W50", "W51", "W52", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W1042"]}

✅ R10 executed run 5 graded by code

W50, W51, W52, W337, W338, W396, W546, W682, W801, W895, W1042. These cite W40, W41, W229, W602, W660 or W895, which cite W01.

JSON: {"works": ["W50", "W51", "W52", "W337", "W338", "W396", "W546", "W682", "W801", "W895", "W1042"]}