Q20, run 2026-11-06-0206-sonnet

← back to the run

Which works in the data are preprints of another work that is also in the data?

Tests: Versions: a shared title is the clue, version_of the evidence; W18 is a preprint with no published version.

Expected: works = ['W02', 'W1049', 'W210', 'W219', 'W226', 'W263', 'W322', 'W407', 'W416', 'W42', 'W432', 'W460', 'W506', 'W522', 'W524', 'W545', 'W556', 'W611', 'W612', 'W663', 'W728', 'W730', 'W877']

✅ R1 csv run 1 graded by code

23 works: W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877, W1049. W18, W221, W306, W336, W566, W745, W747, W780, W781, W824 and W1054 have no `version_of`.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R1 csv run 2 graded by code

23 works: W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877 and W1049. Each has a `version_of` that exists in the data.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R1 csv run 3 graded by code

23 preprints have a `version_of` target that is in the data. They are listed in the JSON.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R1 csv run 4 graded by code

23 preprints with an existing parent, listed in the JSON. PPR records without `version_of` are excluded.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R1 csv run 5 graded by code

W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877, W1049. Preprints with no `version_of` target are not listed.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R3 csv+padding run 1 graded by code

W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877, W1049.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R3 csv+padding run 2 graded by code

23 preprints, listed in the JSON.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R3 csv+padding run 3 graded by code

23 preprints whose `version_of` target is in the data. They are listed in the JSON, from W02 → W03 through W1049 → W789.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R3 csv+padding run 4 graded by code

23 preprints, listed in the JSON. Each has a `version_of` target that is present in works.csv.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R3 csv+padding run 5 graded by code

The 23 preprints listed in the JSON, each pointing to a work in the data through `version_of`.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R4 csv+markdown run 1 graded by code

23 preprints have their published version in the data (listed in the JSON).

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R4 csv+markdown run 2 graded by code

23 works: W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877 and W1049.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R4 csv+markdown run 3 graded by code

23 preprints have their published version in the data. The IDs are in the JSON below.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R4 csv+markdown run 4 graded by code

W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877 and W1049.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R4 csv+markdown run 5 graded by code

The 23 preprints are listed in the JSON.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R5 csv+ontology run 1 graded by code

23 preprints: W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877 and W1049.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R5 csv+ontology run 2 graded by code

W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877, W1049.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R5 csv+ontology run 3 graded by code

23 preprints whose published version is in the data: W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877 and W1049.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R5 csv+ontology run 4 graded by code

The preprints with a published version in the data are W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877 and W1049.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

✅ R5 csv+ontology run 5 graded by code

W02, W42, W210, W219, W226, W263, W322, W407, W416, W432, W460, W506, W522, W524, W545, W556, W611, W612, W663, W728, W730, W877 and W1049. Each has a version_of target that is present in the data.

JSON: {"works": ["W02", "W42", "W210", "W219", "W226", "W263", "W322", "W407", "W416", "W432", "W460", "W506", "W522", "W524", "W545", "W556", "W611", "W612", "W663", "W728", "W730", "W877", "W1049"]}

❌ R10 executed run 1 graded by code

None. The unmerged PPR works (W18, W221, W306, W336, W566, W745, W747, W780, W781, W824, W1054) have no published counterpart in the data.

JSON: {"works": []}

Why: works: expected ['W02', 'W1049', 'W210', 'W219', 'W226', 'W263', 'W322', 'W407', 'W416', 'W42', 'W432', 'W460', 'W506', 'W522', 'W524', 'W545', 'W556', 'W611', 'W612', 'W663', 'W728', 'W730', 'W877'], got []

❌ R10 executed run 2 graded by code

None. Preprints W02, W219 and similar are already merged into their published versions, and the remaining PPR works have no published counterpart in the data.

JSON: {"works": []}

Why: works: expected ['W02', 'W1049', 'W210', 'W219', 'W226', 'W263', 'W322', 'W407', 'W416', 'W42', 'W432', 'W460', 'W506', 'W522', 'W524', 'W545', 'W556', 'W611', 'W612', 'W663', 'W728', 'W730', 'W877'], got []

❌ R10 executed run 3 graded by code

None. Preprints are merged into their published versions, and none of the standalone PPR records matches another work in the data.

JSON: {"works": []}

Why: works: expected ['W02', 'W1049', 'W210', 'W219', 'W226', 'W263', 'W322', 'W407', 'W416', 'W42', 'W432', 'W460', 'W506', 'W522', 'W524', 'W545', 'W556', 'W611', 'W612', 'W663', 'W728', 'W730', 'W877'], got []

❌ R10 executed run 4 graded by code

None. The merged preprint records (W02, W42 and others) are folded into their published works, and the PPR works that remain have no published version in the data.

JSON: {"works": []}

Why: works: expected ['W02', 'W1049', 'W210', 'W219', 'W226', 'W263', 'W322', 'W407', 'W416', 'W42', 'W432', 'W460', 'W506', 'W522', 'W524', 'W545', 'W556', 'W611', 'W612', 'W663', 'W728', 'W730', 'W877'], got []

❌ R10 executed run 5 graded by code

None. The preprint records were merged into their published versions, and the standalone preprints (W18, W221, W306 and similar) have no linked published version in the data.

JSON: {"works": []}

Why: works: expected ['W02', 'W1049', 'W210', 'W219', 'W226', 'W263', 'W322', 'W407', 'W416', 'W42', 'W432', 'W460', 'W506', 'W522', 'W524', 'W545', 'W556', 'W611', 'W612', 'W663', 'W728', 'W730', 'W877'], got []