Research analytics: questions and traps

Each question looks like a lookup, and each has an obvious wrong answer. Only the question text is sent to the model, which must give its answer as a typed JSON field, except Q47 and Q48, which ask why and are graded against a checklist. The trap and the key are never sent. Keys are computed from the data, and each is checked against one written by hand, so the two can't silently disagree.

The implicit meanings

Implicit meaning	What it says	How the data sets it	Questions
Who is who	One person can have several author records, matched by ORCID; one name can belong to different people	A01, A02 and A03 are Mara Ellison (one ORCID); A04, also "M. Ellison", is someone else at Kesting; A05 and A06 are two Jonas Brandts. By record, Priya Ramanathan (A07, 4 works) beats each Mara record; merged, Mara has 5 (Q6). Q7 combines identity, versions, document types and the delisted source	Q1 to Q7
Institution hierarchy	A lab's works are also its centre's, its university's and its system's, each work counted once	Ultimate parent three levels up (Q8); a university's count includes its labs (Q9); a system's count includes both universities, with W12 shared between them counted once (Q10)	Q8 to Q11 (also Q38, Q44, Q46)
Counting method	Full counting gives each institution on a work the whole work; fractional counting splits it by author. The two can rank institutions differently	Kesting 2024: full 3, fractional 2. Rostval 2024: full 4, fractional 1 (four works, one author of four from Rostval on each), so the ranking flips (Q13). GB 2024: full 7, fractional 6. All fractional answers are whole numbers by design	Q12 to Q14, Q47
Document types	Editorials, errata and front matter aren't research output	Editorials (W04, W76), an erratum (W05), paratext (W09, W10)	Q15 to Q17
Versions	A preprint and its published version are one work, and a citation of either is a citation of that work	W02 is the preprint of W03 and W42 of W31 (`version_of`); W18 is a preprint with no published version; W15 cites both W02 and W03	Q18 to Q21
Citation windows	A citation's year is the citing work's year, and an older work has had longer to be cited	W30 (2019) has 5 citations in all but 1 within its year plus two; W31 (2023) has 4 in all, all within the window	Q22, Q23
Retractions	A retracted work isn't research output, and its citations don't count	W23 (Lena Vossberg, Kesting) is retracted and cited once (W39)	Q24 to Q26
Citations at several levels	Citations roll up to institutions, each citing work counted once, and self-citation means something different for an author, an institution and a country	Kesting is cited by three single-author Lorvain works and by one Rostval work with four Rostval authors (counted per author, Rostval would win: Q27, Q28). The sonar paper (W02/W03) is cited by 7 works: author self-citations 2, University of Albury 2, the system 3, GB 4 (Q29, Q30)	Q27 to Q30
Citation chains	A work two hops away cites a work that cites the first, and each is counted once	Two hops from W01: W40 and W41, then W50, W51 and W52, with W51 citing both and a test record W90 left out	Q31, Q32

Q7, Q9, Q11, Q16 and Q25 also depend on the local rules: they combine several traps.

The local meanings

Each rule is stated only in the ontology, and none can be guessed from a code's name or from general conventions.

Local rule	How the data sets it	Where the ontology states it	Questions
`STG` records are test data: they and their citations are ignored	Works with `record_source` STG (W90 to W92, at Kesting, by A21) describe no real work: they're ignored, with their authorships and the citations they make. With them, W22 would be Kesting's most-cited work (4 citations against W20's 3)	`res:origin-STG`; `res:Work` (condition 1); `res:Citation`	Q33, Q34 (also Q11, Q16, Q25, Q31)
`SUP` is a supplement article, which is research output, not supplementary material	A peer-reviewed article in a supplement issue, counted like an article (W17, W24)	`res:wtype-SUP`; `res:Work` (condition 2)	Q34 to Q36
When an author lists several affiliations on a work, only the one listed last is credited	Only the one with the highest `affil_seq` counts. W70 lists Tessel and then Halloran, so it goes to Halloran; W71 lists Halloran and then Tessel, so it goes to Tessel	`res:affiliationSeq` (also `res:Authorship`, `res:country`)	Q37, Q38, Q45 (also Q9, Q48)
The tiers count upwards: `K3` is the highest, `K1` the lowest	The reverse of the natural guess, where tier 1 would be the best	`res:tier`, `res:tier-K3` to `res:tier-K1`	Q39 to Q41
`K0` isn't a tier but a delisting: works in a delisted source aren't research output	Works in a K0 source (W07, W75) are left out of every count	`res:tier-K0`; `res:Work` (condition 4)	Q42, Q43 (also Q7)
An affiliated institute (`AFF`) is not a unit of its parent: its parent link is an agreement, not membership	Halloran's parent link records an agreement, so its works are never credited to the University of Albury or the system, and its ultimate parent is itself	`res:itype-AFF`; `res:Institution`; `res:parent`	Q44 to Q46 (also Q9, Q48)

"Research work" is the dataset's own term, so some answers given without the ontology, such as counting editorials, could be called a reasonable reading rather than an error. A review found they're mostly the questions that depend on the local rules anyway (see the conclusions).

Q1How many distinct people do the author records named "Mara Ellison", "M. Ellison" and "Ellison, Mara J." stand for?

The trap: Identity. A01, A02 and A03 share an ORCID; A04, also 'M. Ellison', has another.

Expected: count 2 · graded by code

Q2List every work, of any type, that Mara Ellison (A01) is an author of.

The trap: Identity. Her works are spread over three author records, and A04's works (W08, W16) are not hers.

Expected: works ['W01', 'W02', 'W03', 'W04', 'W05', 'W06', 'W07', 'W12', 'W19'] · graded by code

Q3Is the Jonas Brandt who wrote W37 the same person as the Jonas Brandt who wrote W38?

The trap: Identity. The same name, different ORCIDs.

Expected: same no · graded by code

Q4How many research works has the Jonas Brandt of Albury Metropolitan University (A05) authored?

The trap: Identity. The other Jonas Brandt's works (W28, W34, W38) are not his.

Expected: count 2 · graded by code

Q5Is the M. Ellison who wrote W08 the same person as the Mara Ellison who wrote W01?

The trap: Identity. A name variant shared with a different person.

Expected: same no · graded by code

Q6Which author has the most research works credited to the University of Albury, its units included?

The trap: Identity. Mara Ellison's five works are split over three records; by record, Priya Ramanathan (four) would come first.

Expected: author A01 · graded by code

Q7How many research works has Mara Ellison authored?

The trap: Everything at once. Identity, versions (W02 is W03), document types (W04, W05) and the delisted source (W07).

Expected: count 5 · graded by code

Q8What is the ultimate parent of the Tessel Laboratory for Ocean Sensing (I04)?

The trap: Hierarchy. Three levels up, lab to centre to university to system.

Expected: parent I01 · graded by code

Q9How many research works is the University of Albury (I02) credited with, its units included?

The trap: Hierarchy. The labs' works count, each work once; also versions, types, the affiliation rule and the affiliated institute.

Expected: count 8 · graded by code

Q10How many research works is the Albury University System (I01) credited with, all its units included?

The trap: Hierarchy. Both universities and their labs, a work shared between them (W12) counted once.

Expected: count 12 · graded by code

Q11How many research works is Kesting University (I07) credited with, its units included?

The trap: Hierarchy with retraction and test records. The institute's W24 counts; W23 (retracted) and W90 to W92 (STG) don't.

Expected: count 6 · graded by code

Q12How many research works did Kesting University (I07, units included) publish in 2024, under full counting and under fractional counting (each work split equally among its authors)?

The trap: Counting method. W20 is half Kesting's, W21 two quarters, W22 whole.

Expected: full 3, fractional 2 · graded by code

Q13In 2024, which had more research works, Kesting University (I07) or Rostval Technical University (I09): under full counting, and under fractional counting?

The trap: Counting method. The ranking flips.

Expected: full I09, fractional I07 · graded by code

Q14How many research works did GB institutions publish in 2024, under full counting and under fractional counting (each work split equally among its authors, each author's share going to their institution's country)?

The trap: Counting method at country level..

Expected: full 7, fractional 6 · graded by code

Q15How many research works appeared in Ocean Sensing Letters (S02) in 2024?

The trap: Document types. The erratum W05 and the front matter W10 are not research output.

Expected: count 4 · graded by code

Q16How many research works appeared in the Journal of Marine Autonomy (S01)?

The trap: Document types. An editorial (W04) and paratext (W09) are not research output; also a test record (W91).

Expected: count 10 · graded by code

Q17Which of Mara Ellison's works are not research output because of their document type?

The trap: Document types. An editorial and an erratum, under two author records.

Expected: works ['W04', 'W05'] · graded by code

Q18How many distinct research works has Priya Ramanathan (A07) authored?

The trap: Versions. W02 is the preprint of W03; also the erratum W05 is not a work of research.

Expected: count 4 · graded by code

Q19How many distinct works cite "Drift-aware mapping with low-cost sonar", in either of its versions?

The trap: Versions. Citations to the preprint W02 and to W03 are citations of one work; W15 cites both and counts once.

Expected: count 7 · graded by code

Q20Which works in the data are preprints of another work that is also in the data?

The trap: Versions. A shared title is the clue, version_of the evidence; W18 is a preprint with no published version.

Expected: works ['W02', 'W42'] · graded by code

Q21How many research works is Oskel Polytechnic (I14) credited with?

The trap: Versions. W42 is the preprint of W31.

Expected: count 4 · graded by code

Q22Which has more citations, W30 or W31 (counting both versions of each): in total, and within its publication year and the two years after?

The trap: Citation windows. The older work has had more time; over the same span the newer one leads.

Expected: more_in_total W30, more_in_window W31 · graded by code

Q23How many citations has W30 received in total, and how many within its publication year and the two years after?

The trap: Citation windows. A citation's year is the citing work's year.

Expected: total 5, in_window 1 · graded by code

Q24How many research works has Lena Vossberg (A09) authored?

The trap: Retractions. W23 is retracted.

Expected: count 3 · graded by code

Q25What is the total of the citation counts of Lena Vossberg's research works?

The trap: Retractions and test records. W23's citation (W39) doesn't count, nor do the STG works citing W22.

Expected: count 6 · graded by code

Q26Does W23 count towards Kesting University's research output?

The trap: Retractions. A retracted work is not research output.

Expected: counts no · graded by code

Q27Which other institution's works cite Kesting University's research works the most, each citing work counted once per institution, and how many of its works do?

The trap: Citations rolled up. Rostval's one citing work has four Rostval authors; counted per author it would win.

Expected: institution I12, citing_works 3 · graded by code

Q28How many distinct works from Rostval Technical University (I09) cite Kesting University's research works?

The trap: Citations rolled up. A work counts once, not once per author.

Expected: count 1 · graded by code

Q29Of the works citing "Drift-aware mapping with low-cost sonar" (either version), how many are self-citations at the level of authors, of the University of Albury (units included), and of countries?

The trap: Self-citation depends on the level; W12 is Mara Ellison under another record, W13 her co-author; W16 is the other M. Ellison..

Expected: author 2, institution 2, country 4 · graded by code

Q30Of the works citing "Drift-aware mapping with low-cost sonar" (either version), how many are self-citations at the level of the Albury University System?

The trap: Self-citation at a level between institution and country..

Expected: count 3 · graded by code

Q31Which works cite at least one of the works that cite W01?

The trap: Two hops; W51 cites both first-hop works; W90 is a test record..

Expected: works ['W50', 'W51', 'W52'] · graded by code

Q32How many distinct works are within two citation hops of W01 (citing it, or citing a work that cites it)?

The trap: Two hops, counted once each..

Expected: count 5 · graded by code

Q33Which of Kesting University's research works has the most citations?

The trap: Local rule. STG records are test data; with them W22 would lead.

Expected: work W20 · graded by code

Q34How many research works was Kesting University (units included) credited with in 2025?

The trap: Local rules. STG records are test data; SUP is a research article.

Expected: count 3 · graded by code

Q35How many research works is the Kesting Institute of Hydrodynamics (I08) credited with?

The trap: Local code. SUP is a supplement article, research output, not supplementary material.

Expected: count 2 · graded by code

Q36How many research works appeared in the Proceedings of the Hydrodynamics Forum (S06)?

The trap: Local code. Both SUP works count; the foreword (EDT) doesn't.

Expected: count 2 · graded by code

Q37Which institution is W70 credited to?

The trap: Local rule. Only the affiliation listed last counts.

Expected: institution I06 · graded by code

Q38Which institutions is W71 credited to, at every level?

The trap: Local rule on affiliations, then the hierarchy. The Tessel Laboratory is listed last, so Halloran gets nothing.

Expected: institutions ['I01', 'I02', 'I03', 'I04'] · graded by code

Q39How many of the University of Albury's research works (units included) appeared in a source of the highest tier?

The trap: Local code. The tiers count upwards, K3 the highest.

Expected: count 3 · graded by code

Q40Which source is in the highest tier?

The trap: Local code. K3 is the highest tier, not K1.

Expected: source S01 · graded by code

Q41Is Coastal Engineering Review (S03) in a higher tier than Ocean Sensing Letters (S02)?

The trap: Local code. K1 is below K2.

Expected: higher no · graded by code

Q42How many research works appeared in Global Marine Science Reports (S04)?

The trap: Local code. K0 is a delisted source, whose works are not research output.

Expected: count 0 · graded by code

Q43Is W07 research output?

The trap: Local code. An article in a delisted source.

Expected: research no · graded by code

Q44Which institutions are part of the Albury University System, at any level below it?

The trap: Local type. An affiliated institute (AFF) is not a unit of its parent.

Expected: institutions ['I02', 'I03', 'I04', 'I05'] · graded by code

Q45How many research works is the Halloran Institute for Coastal Studies (I06) credited with?

The trap: Local rule on affiliations. W71 lists Halloran first, so it goes to the Tessel Laboratory.

Expected: count 3 · graded by code

Q46What is the ultimate parent of the Halloran Institute for Coastal Studies (I06)?

The trap: Local type. An affiliated institute's parent link is an agreement, not membership.

Expected: parent I06 · graded by code

Q47Why do Kesting University and Rostval Technical University swap places in 2024 between full and fractional counting?

The trap: Counting method. Why the ranking flips.

Graded against a checklist:

Says each of Rostval's four 2024 works has four authors with only one from Rostval, so a quarter each
Says Kesting holds larger shares: a single-author review (W22) and half of W20 and W21

Q48Why is W70 not one of the University of Albury's research works, though its author lists the Tessel Laboratory?

The trap: Local rules. The last-listed affiliation, and affiliated institutes.

Graded against a checklist:

Says only the affiliation listed last counts, so W70 is credited to the Halloran Institute
Says the Halloran Institute is an affiliated institute, not a unit, so its works don't roll up to the University of Albury