Q17, run 2026-11-06-0206-haiku

← back to the run

Which of Mara Ellison's works are not research output because of their document type?

Tests: Document types: an editorial and an erratum, under two author records.

Expected: works = ['W04', 'W05', 'W271', 'W813', 'W936', 'W949']

❌ R1 csv run 1 graded by code

W04, W05, W271, W813, W949. These are EDT or ERR records.

JSON: {"works": ["W04", "W05", "W271", "W813", "W949"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271', 'W813', 'W949']

❌ R1 csv run 2 graded by code

W04, W05 and W271. These are editorial (EDT) and erratum (ERR) records.

JSON: {"works": ["W04", "W05", "W271"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271']

❌ R1 csv run 3 graded by code

W04 and W271 (EDT), W05, W813 and W949 (ERR).

JSON: {"works": ["W04", "W05", "W271", "W813", "W949"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271', 'W813', 'W949']

❌ R1 csv run 4 graded by code

W04 (EDT), W05 (ERR), W271 (EDT), W813 (ERR) and W949 (ERR) are not research output because of their document type.

JSON: {"works": ["W04", "W05", "W271", "W813", "W949"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271', 'W813', 'W949']

❌ R1 csv run 5 graded by code

(no answer found)

JSON: {"works": ["W04", "W05", "W271", "W949"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271', 'W949']

❌ R3 csv+padding run 1 graded by code

W04 (EDT), W05 (ERR), W271 (EDT), W813 (ERR) and W949 (ERR).

JSON: {"works": ["W04", "W05", "W271", "W813", "W949"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271', 'W813', 'W949']

❌ R3 csv+padding run 2 graded by code

W04, W05, W271, W813, W949 (two editorials and three errata).

JSON: {"works": ["W04", "W05", "W271", "W813", "W949"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271', 'W813', 'W949']

❌ R3 csv+padding run 3 graded by code

W04 and W271 (editorials), W05 and W813 and W949 (errata).

JSON: {"works": ["W04", "W05", "W271", "W813", "W949"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271', 'W813', 'W949']

❌ R3 csv+padding run 4 graded by code

W04 and W271 (EDT), W05 and W949 (ERR).

JSON: {"works": ["W04", "W05", "W271", "W949"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271', 'W949']

✅ R3 csv+padding run 5 graded by code

W04, W05, W271, W813, W936, W949. These are an editorial or erratum (EDT or ERR), so they are not research output.

JSON: {"works": ["W04", "W05", "W271", "W813", "W936", "W949"]}

❌ R4 csv+markdown run 1 graded by code

(no answer found)

Why: no structured answer

❌ R4 csv+markdown run 2 graded by code

(no answer found)

Why: no structured answer

✅ R4 csv+markdown run 3 graded by code

W04, W05, W271, W813, W949 and W936 are editorials or errata.

JSON: {"works": ["W04", "W05", "W271", "W813", "W949", "W936"]}

❌ R4 csv+markdown run 4 graded by code

(no answer found)

Why: no structured answer

❌ R4 csv+markdown run 5 graded by code

W04, W05, W271, W936, W949. These are editorials and errata, which are not research output by type.

JSON: {"works": ["W04", "W05", "W271", "W936", "W949"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271', 'W936', 'W949']

❌ R5 csv+ontology run 1 graded by code

(no answer found)

Why: no structured answer

❌ R5 csv+ontology run 2 graded by code

W04, W05, W271. These are the editorial and erratum records of A01.

JSON: {"works": ["W04", "W05", "W271"]}

Why: works: expected ['W04', 'W05', 'W271', 'W813', 'W936', 'W949'], got ['W04', 'W05', 'W271']

✅ R5 csv+ontology run 3 graded by code

W04, W05, W271, W813, W936, W949 (EDT or ERR).

JSON: {"works": ["W04", "W05", "W271", "W813", "W936", "W949"]}

✅ R5 csv+ontology run 4 graded by code

W04, W05, W271, W813, W936, W949. These are the editorials (EDT) and errata (ERR) on that person's record.

JSON: {"works": ["W04", "W05", "W271", "W813", "W936", "W949"]}

✅ R5 csv+ontology run 5 graded by code

W04 and W271 (editorials), W05, W813 and W949 (errata), and W936 (editorial).

JSON: {"works": ["W04", "W05", "W271", "W813", "W936", "W949"]}

✅ R10 executed run 1 graded by code

W04, W05, W271, W813, W936 and W949. Their types are EDT or ERR, which are not research output. W07, W449 and W763 are excluded by source delisting and W374 by retraction, so they are not type-based.

JSON: {"works": ["W04", "W05", "W271", "W813", "W936", "W949"]}

✅ R10 executed run 2 graded by code

W04, W05, W271, W813, W936, W949. These are editorials (EDT) and errata (ERR); W07, W374, W449 and W763 are excluded for other reasons (source, retraction).

JSON: {"works": ["W04", "W05", "W271", "W813", "W936", "W949"]}

✅ R10 executed run 3 graded by code

W04 and W271 (editorials), W05 and W813 and W949 (errata), and W936 (editorial). W07, W449 and W763 are excluded for their delisted source; W374 is retracted.

JSON: {"works": ["W04", "W05", "W271", "W813", "W936", "W949"]}

✅ R10 executed run 4 graded by code

W04, W05, W271, W813, W936, W949.
These are editorials (EDT) or errata (ERR) credited to A01, and so not research output on document type. W07, W449 and W763 are excluded for source or other reasons.

JSON: {"works": ["W04", "W05", "W271", "W813", "W936", "W949"]}

✅ R10 executed run 5 graded by code

W04, W05, W271, W813, W936 and W949 are not research output because they are editorials or errata. W07, W449 and W763 are excluded for a delisted source, and W374 for retraction.

JSON: {"works": ["W04", "W05", "W271", "W813", "W936", "W949"]}