Run 2026-11-06-0855-sonnet-query

Research analytics: at 850 works claude:sonnet query 48 questions started 2026-11-06 08:55 commit b4ebd07 done

Accuracy

Condition	Repeats	Accuracy	95% interval
D6 `sparql+axioms`	5	91%	85–96%

Queries

Condition	Queries run	Failed	Rounds, on average
D6 `sparql+axioms`	174	11	6.6

Question by question

One mark per repeat; hover for the grader's reason, click a question for every answer to it.

Question D6 sparql+axioms

Q1 How many distinct people do the author records named "Mara Ellison", "M. Ellison" and "Ellison, Mara J." stand for? ❌❌❌❌❌

Q2 List every work, of any type, that Mara Ellison (A01) is an author of. ✅✅✅✅❌

Q3 Is the Jonas Brandt who wrote W37 the same person as the Jonas Brandt who wrote W38? ✅✅✅✅✅

Q4 How many research works has the Jonas Brandt of Albury Metropolitan University (A05) authored? ✅✅✅✅✅

Q5 Is the M. Ellison who wrote W08 the same person as the Mara Ellison who wrote W01? ✅✅✅✅✅

Q6 Which author has the most research works credited to the University of Albury, its units included? ✅✅✅✅❌

Q7 How many research works has Mara Ellison authored? ✅✅✅✅✅

Q8 What is the ultimate parent of the Tessel Laboratory for Ocean Sensing (I04)? ✅✅✅✅✅

Q9 How many research works is the University of Albury (I02) credited with, its units included? ✅✅✅✅✅

Q10 How many research works is the Albury University System (I01) credited with, all its units included? ✅✅✅✅✅

Q11 How many research works is Kesting University (I07) credited with, its units included? ✅✅✅✅✅

Q12 How many research works did Kesting University (I07, units included) publish in 2024, under full counting and under fractional counting (each work split equally among its authors)? 🟡✅✅✅✅

Q13 In 2024, which had more research works, Kesting University (I07) or Rostval Technical University (I09): under full counting, and under fractional counting? ✅✅✅✅✅

Q14 How many research works did GB institutions publish in 2024, under full counting and under fractional counting (each work split equally among its authors, each author's share going to their institution's country)? ❌✅✅✅✅

Q15 How many research works appeared in Ocean Sensing Letters (S02) in 2024? ✅✅✅✅✅

Q16 How many research works appeared in the Journal of Marine Autonomy (S01)? ✅✅✅✅✅

Q17 Which of Mara Ellison's works are not research output because of their document type? ✅✅✅✅✅

Q18 How many distinct research works has Priya Ramanathan (A07) authored? ✅✅✅✅✅

Q19 How many distinct works cite "Drift-aware mapping with low-cost sonar", in either of its versions? ❌✅✅✅❌

Q20 Which works in the data are preprints of another work that is also in the data? ✅✅✅✅✅

Q21 How many research works is Oskel Polytechnic (I14) credited with? ✅✅✅✅✅

Q22 Which has more citations, W30 or W31 (counting both versions of each): in total, and within its publication year and the two years after? ✅✅✅✅✅

Q23 How many citations has W30 received in total, and how many within its publication year and the two years after? 🟡✅✅✅✅

Q24 How many research works has Lena Vossberg (A09) authored? ✅✅✅✅✅

Q25 What is the total of the citation counts of Lena Vossberg's research works? ✅✅❌✅✅

Q26 Does W23 count towards Kesting University's research output? ✅✅✅✅✅

Q27 Which other institution's works cite Kesting University's research works the most, each citing work counted once per institution, and how many of its works do? ✅✅✅✅✅

Q28 How many distinct works from Rostval Technical University (I09) cite Kesting University's research works? ✅✅✅✅✅

Q29 Of the works citing "Drift-aware mapping with low-cost sonar" (either version), how many are self-citations at the level of authors, of the University of Albury (units included), and of countries? 🟡✅✅✅✅

Q30 Of the works citing "Drift-aware mapping with low-cost sonar" (either version), how many are self-citations at the level of the Albury University System? ❌✅✅✅✅

Q31 Which works cite at least one of the works that cite W01? ✅✅✅✅❌

Q32 How many distinct works are within two citation hops of W01 (citing it, or citing a work that cites it)? ✅✅✅✅✅

Q33 Which of Kesting University's research works has the most citations? ✅✅✅✅✅

Q34 How many research works was Kesting University (units included) credited with in 2025? ✅✅✅✅✅

Q35 How many research works is the Kesting Institute of Hydrodynamics (I08) credited with? ✅✅✅✅✅

Q36 How many research works appeared in the Proceedings of the Hydrodynamics Forum (S06)? ✅✅✅✅✅

Q37 Which institution is W70 credited to? ✅✅✅✅✅

Q38 Which institutions is W71 credited to, at every level? ✅✅✅✅✅

Q39 How many of the University of Albury's research works (units included) appeared in a source of the highest tier? ❌✅✅✅✅

Q40 Which source is in the highest tier? ✅✅✅✅✅

Q41 Is Coastal Engineering Review (S03) in a higher tier than Ocean Sensing Letters (S02)? ✅✅✅✅✅

Q42 How many research works appeared in Global Marine Science Reports (S04)? ✅✅✅✅✅

Q43 Is W07 research output? ✅✅✅✅✅

Q44 Which institutions are part of the Albury University System, at any level below it? ✅✅✅✅✅

Q45 How many research works is the Halloran Institute for Coastal Studies (I06) credited with? ✅✅✅✅✅

Q46 What is the ultimate parent of the Halloran Institute for Coastal Studies (I06)? ✅✅✅✅✅

Q47 Why do Kesting University and Rostval Technical University swap places in 2024 between full and fractional counting? ❌🟡❌✅✅

Q48 Why is W70 not one of the University of Albury's research works, though its author lists the Tessel Laboratory? ❌🟡🟡🟡🟡

Question	D6 `sparql+axioms`
Q1 How many distinct people do the author records named "Mara Ellison", "M. Ellison" and "Ellison, Mara J." stand for?	❌❌❌❌❌
Q2 List every work, of any type, that Mara Ellison (A01) is an author of.	✅✅✅✅❌
Q3 Is the Jonas Brandt who wrote W37 the same person as the Jonas Brandt who wrote W38?	✅✅✅✅✅
Q4 How many research works has the Jonas Brandt of Albury Metropolitan University (A05) authored?	✅✅✅✅✅
Q5 Is the M. Ellison who wrote W08 the same person as the Mara Ellison who wrote W01?	✅✅✅✅✅
Q6 Which author has the most research works credited to the University of Albury, its units included?	✅✅✅✅❌
Q7 How many research works has Mara Ellison authored?	✅✅✅✅✅
Q8 What is the ultimate parent of the Tessel Laboratory for Ocean Sensing (I04)?	✅✅✅✅✅
Q9 How many research works is the University of Albury (I02) credited with, its units included?	✅✅✅✅✅
Q10 How many research works is the Albury University System (I01) credited with, all its units included?	✅✅✅✅✅
Q11 How many research works is Kesting University (I07) credited with, its units included?	✅✅✅✅✅
Q12 How many research works did Kesting University (I07, units included) publish in 2024, under full counting and under fractional counting (each work split equally among its authors)?	🟡✅✅✅✅
Q13 In 2024, which had more research works, Kesting University (I07) or Rostval Technical University (I09): under full counting, and under fractional counting?	✅✅✅✅✅
Q14 How many research works did GB institutions publish in 2024, under full counting and under fractional counting (each work split equally among its authors, each author's share going to their institution's country)?	❌✅✅✅✅
Q15 How many research works appeared in Ocean Sensing Letters (S02) in 2024?	✅✅✅✅✅
Q16 How many research works appeared in the Journal of Marine Autonomy (S01)?	✅✅✅✅✅
Q17 Which of Mara Ellison's works are not research output because of their document type?	✅✅✅✅✅
Q18 How many distinct research works has Priya Ramanathan (A07) authored?	✅✅✅✅✅
Q19 How many distinct works cite "Drift-aware mapping with low-cost sonar", in either of its versions?	❌✅✅✅❌
Q20 Which works in the data are preprints of another work that is also in the data?	✅✅✅✅✅
Q21 How many research works is Oskel Polytechnic (I14) credited with?	✅✅✅✅✅
Q22 Which has more citations, W30 or W31 (counting both versions of each): in total, and within its publication year and the two years after?	✅✅✅✅✅
Q23 How many citations has W30 received in total, and how many within its publication year and the two years after?	🟡✅✅✅✅
Q24 How many research works has Lena Vossberg (A09) authored?	✅✅✅✅✅
Q25 What is the total of the citation counts of Lena Vossberg's research works?	✅✅❌✅✅
Q26 Does W23 count towards Kesting University's research output?	✅✅✅✅✅
Q27 Which other institution's works cite Kesting University's research works the most, each citing work counted once per institution, and how many of its works do?	✅✅✅✅✅
Q28 How many distinct works from Rostval Technical University (I09) cite Kesting University's research works?	✅✅✅✅✅
Q29 Of the works citing "Drift-aware mapping with low-cost sonar" (either version), how many are self-citations at the level of authors, of the University of Albury (units included), and of countries?	🟡✅✅✅✅
Q30 Of the works citing "Drift-aware mapping with low-cost sonar" (either version), how many are self-citations at the level of the Albury University System?	❌✅✅✅✅
Q31 Which works cite at least one of the works that cite W01?	✅✅✅✅❌
Q32 How many distinct works are within two citation hops of W01 (citing it, or citing a work that cites it)?	✅✅✅✅✅
Q33 Which of Kesting University's research works has the most citations?	✅✅✅✅✅
Q34 How many research works was Kesting University (units included) credited with in 2025?	✅✅✅✅✅
Q35 How many research works is the Kesting Institute of Hydrodynamics (I08) credited with?	✅✅✅✅✅
Q36 How many research works appeared in the Proceedings of the Hydrodynamics Forum (S06)?	✅✅✅✅✅
Q37 Which institution is W70 credited to?	✅✅✅✅✅
Q38 Which institutions is W71 credited to, at every level?	✅✅✅✅✅
Q39 How many of the University of Albury's research works (units included) appeared in a source of the highest tier?	❌✅✅✅✅
Q40 Which source is in the highest tier?	✅✅✅✅✅
Q41 Is Coastal Engineering Review (S03) in a higher tier than Ocean Sensing Letters (S02)?	✅✅✅✅✅
Q42 How many research works appeared in Global Marine Science Reports (S04)?	✅✅✅✅✅
Q43 Is W07 research output?	✅✅✅✅✅
Q44 Which institutions are part of the Albury University System, at any level below it?	✅✅✅✅✅
Q45 How many research works is the Halloran Institute for Coastal Studies (I06) credited with?	✅✅✅✅✅
Q46 What is the ultimate parent of the Halloran Institute for Coastal Studies (I06)?	✅✅✅✅✅
Q47 Why do Kesting University and Rostval Technical University swap places in 2024 between full and fractional counting?	❌🟡❌✅✅
Q48 Why is W70 not one of the University of Albury's research works, though its author lists the Tessel Laboratory?	❌🟡🟡🟡🟡

The prompts

The report, as report.md

research_850: claude:sonnet, 2026-11-06 08:55

48 questions, 5 repeats per condition. Correct = 1, partial = ½. Graded by code where a question has a typed key, by claude:sonnet with a checklist otherwise.

Accuracy

Condition	Accuracy	95% interval	Questions unstable across repeats
D6 `sparql+axioms`	91%	85% to 96%	13 of 48

By question

Question	Graded by	D6 `sparql+axioms`
Q1	code	❌ ❌ ❌ ❌ ❌
Q2	code	✅ ✅ ✅ ✅ ❌
Q3	code	✅ ✅ ✅ ✅ ✅
Q4	code	✅ ✅ ✅ ✅ ✅
Q5	code	✅ ✅ ✅ ✅ ✅
Q6	code	✅ ✅ ✅ ✅ ❌
Q7	code	✅ ✅ ✅ ✅ ✅
Q8	code	✅ ✅ ✅ ✅ ✅
Q9	code	✅ ✅ ✅ ✅ ✅
Q10	code	✅ ✅ ✅ ✅ ✅
Q11	code	✅ ✅ ✅ ✅ ✅
Q12	code	🟡 ✅ ✅ ✅ ✅
Q13	code	✅ ✅ ✅ ✅ ✅
Q14	code	❌ ✅ ✅ ✅ ✅
Q15	code	✅ ✅ ✅ ✅ ✅
Q16	code	✅ ✅ ✅ ✅ ✅
Q17	code	✅ ✅ ✅ ✅ ✅
Q18	code	✅ ✅ ✅ ✅ ✅
Q19	code	❌ ✅ ✅ ✅ ❌
Q20	code	✅ ✅ ✅ ✅ ✅
Q21	code	✅ ✅ ✅ ✅ ✅
Q22	code	✅ ✅ ✅ ✅ ✅
Q23	code	🟡 ✅ ✅ ✅ ✅
Q24	code	✅ ✅ ✅ ✅ ✅
Q25	code	✅ ✅ ❌ ✅ ✅
Q26	code	✅ ✅ ✅ ✅ ✅
Q27	code	✅ ✅ ✅ ✅ ✅
Q28	code	✅ ✅ ✅ ✅ ✅
Q29	code	🟡 ✅ ✅ ✅ ✅
Q30	code	❌ ✅ ✅ ✅ ✅
Q31	code	✅ ✅ ✅ ✅ ❌
Q32	code	✅ ✅ ✅ ✅ ✅
Q33	code	✅ ✅ ✅ ✅ ✅
Q34	code	✅ ✅ ✅ ✅ ✅
Q35	code	✅ ✅ ✅ ✅ ✅
Q36	code	✅ ✅ ✅ ✅ ✅
Q37	code	✅ ✅ ✅ ✅ ✅
Q38	code	✅ ✅ ✅ ✅ ✅
Q39	code	❌ ✅ ✅ ✅ ✅
Q40	code	✅ ✅ ✅ ✅ ✅
Q41	code	✅ ✅ ✅ ✅ ✅
Q42	code	✅ ✅ ✅ ✅ ✅
Q43	code	✅ ✅ ✅ ✅ ✅
Q44	code	✅ ✅ ✅ ✅ ✅
Q45	code	✅ ✅ ✅ ✅ ✅
Q46	code	✅ ✅ ✅ ✅ ✅
Q47	llm	❌ 🟡 ❌ ✅ ✅
Q48	llm	❌ 🟡 🟡 🟡 🟡

Lists, with partial credit

The 6 questions answered with a list of entities: strict accuracy (the whole list right) against the mean F1 of the lists given, which credits a list that is nearly right.

Condition	Strict	Mean F1
D6 `sparql+axioms`	93%	100%

The checklist grader agreed with itself on 9 of 10 answers graded twice.

Why each miss was marked

Q1. How many distinct people do the author records named "Mara Ellison", "M. Ellison" and "Ellison, Mara J." stand for?

D6 sparql+axioms run 1: count: expected 3, got 1
D6 sparql+axioms run 2: count: expected 3, got 1
D6 sparql+axioms run 3: count: expected 3, got 1
D6 sparql+axioms run 4: count: expected 3, got 1
D6 sparql+axioms run 5: count: expected 3, got 1

Q2. List every work, of any type, that Mara Ellison (A01) is an author of.

D6 sparql+axioms run 5: works: expected ['W01', 'W02', 'W03', 'W04', 'W05', 'W06', 'W07', 'W1005', 'W1048', 'W1050', 'W12', 'W19', 'W206', 'W238', 'W271', 'W374', 'W449', 'W464', 'W555', 'W612', 'W663', 'W676', 'W763', 'W813', 'W817', 'W843', 'W869', 'W905', 'W936', 'W949', 'W972', 'W984'], got ['W01', 'W02', 'W03', 'W04', 'W05', 'W06', 'W07', 'W12', 'W19', 'W206', 'W238', 'W271', 'W374', 'W449', 'W464', 'W555', 'W676', 'W763', 'W813', 'W817', 'W843', 'W869', 'W905', 'W936', 'W949', 'W972', 'W984', 'W1005', 'W1048', 'W1050']

Q6. Which author has the most research works credited to the University of Albury, its units included?

D6 sparql+axioms run 5: author: expected A01, got A07

Q12. How many research works did Kesting University (I07, units included) publish in 2024, under full counting and under fractional counting (each work split equally among its authors)?

D6 sparql+axioms run 1: fractional: expected 11, got None

Q14. How many research works did GB institutions publish in 2024, under full counting and under fractional counting (each work split equally among its authors, each author's share going to their institution's country)?

D6 sparql+axioms run 1: full: expected 46, got None; fractional: expected 42, got None

Q19. How many distinct works cite "Drift-aware mapping with low-cost sonar", in either of its versions?

D6 sparql+axioms run 1: count: expected 27, got 30
D6 sparql+axioms run 5: count: expected 27, got 30

Q23. How many citations has W30 received in total, and how many within its publication year and the two years after?

D6 sparql+axioms run 1: total: expected 16, got 18

Q25. What is the total of the citation counts of Lena Vossberg's research works?

D6 sparql+axioms run 3: count: expected 20, got 19

Q29. Of the works citing "Drift-aware mapping with low-cost sonar" (either version), how many are self-citations at the level of authors, of the University of Albury (units included), and of countries?

D6 sparql+axioms run 1: institution: expected 10, got 11; country: expected 20, got 21

Q30. Of the works citing "Drift-aware mapping with low-cost sonar" (either version), how many are self-citations at the level of the Albury University System?

D6 sparql+axioms run 1: count: expected 14, got 15

Q31. Which works cite at least one of the works that cite W01?

D6 sparql+axioms run 5: works: expected ['W1042', 'W337', 'W338', 'W396', 'W50', 'W51', 'W52', 'W546', 'W682', 'W801', 'W895'], got ['W50', 'W51', 'W52', 'W337', 'W338', 'W396', 'W546', 'W682', 'W801', 'W1042']

Q39. How many of the University of Albury's research works (units included) appeared in a source of the highest tier?

D6 sparql+axioms run 1: count: expected 19, got None

Q47. Why do Kesting University and Rostval Technical University swap places in 2024 between full and fractional counting?

D6 sparql+axioms run 1: No answer given; meets none of the checklist points
D6 sparql+axioms run 2: Gives the co-author dilution for I09 (Rostval) but doesn't say Rostval gets only one author's share, and omits Kesting's larger shares (sole or few co-authors); also admits not querying it.
D6 sparql+axioms run 3: Explicitly unanswered; gives only generic counting definitions, none of the checklist points about Rostval's small shares or Kesting's larger shares.

Q48. Why is W70 not one of the University of Albury's research works, though its author lists the Tessel Laboratory?

D6 sparql+axioms run 1: No answer given; meets none of the checklist points
D6 sparql+axioms run 2: Says I06 is not a unit of I02 so crediting it doesn't credit I02, but omits that only the last-listed affiliation counts.
D6 sparql+axioms run 3: Says W70 is credited only to I06, an affiliated institute whose works don't go to the parent (second point), but omits that only the last-listed affiliation counts.
D6 sparql+axioms run 4: Says the work is credited to I06 only, which counts only toward itself and not toward the University of Albury. It never states the last-listed-affiliation rule, and it doesn't say outright that the Halloran Institute is an affiliated institute rather than a unit.
D6 sparql+axioms run 5: Says I06 doesn't count toward I02, but never gives the rule that only the last-listed affiliation counts, and doesn't say Halloran is an affiliated institute rather than a unit.