Q47, run 2026-11-06-0206-haiku

← back to the run

Why do Kesting University and Rostval Technical University swap places in 2024 between full and fractional counting?

Tests: Counting method: why the ranking flips.

Expected: Rostval is on 16 works in 2024, but on every one of them it has at most a quarter of the authors, so its shares add up to 4. Kesting is on 13, 8 of them by Kesting authors alone, so its shares add up to 11.

❌ R1 csv run 1 graded by llm

The premise is wrong. Kesting stays ahead of Rostval in 2024 under both counting methods, so nothing swaps.

Why: Rejects the premise and gives none of the checklist points (Rostval's small shares of multi-author works, Kesting's larger shares).

❌ R1 csv run 2 graded by llm

Not computed yet. This is deferred to the next batch.

Why: No answer given; defers computation and states neither Rostval's small shares nor Kesting's larger shares.

🟡 R1 csv run 3 graded by llm

The premise doesn't hold: the two do not swap. Full counting ties at 14 each. Under fractional counting, Kesting leads 12.0 to 3.5, because Rostval's 2024 works are four-author papers with one Rostval author each.

Why: Covers Rostval's small shares (four-author papers, one Rostval author each), but never explains Kesting's larger shares, and it rejects the swap premise.

❌ R1 csv run 4 graded by llm

(no answer found)

Why: No answer given; meets none of the checklist points

❌ R1 csv run 5 graded by llm

The premise is false; they do not swap. In 2024 Kesting leads both ways: 15 vs 10 under full counting and about 13 vs 2.5 under fractional counting.

Why: Rejects the premise and gives no explanation of Rostval's small fractional shares or Kesting's larger shares; meets neither checklist point.

✅ R3 csv+padding run 1 graded by llm

Rostval's 2024 works mostly have four authors, with only one Rostval author each, so each contributes 0.25 fractional credit (about 4.05 in total). Kesting's 2024 works are mostly 1–2 author papers, so its fractional share is larger (about 6.7). Full counting gives Rostval 14 and Kesting 9, so the rankings swap between the two methods.

🟡 R3 csv+padding run 2 graded by llm

Under full counting, Rostval's 2024 papers are mostly four-author papers with one Rostval author each, giving 16 papers in total. Kesting's papers have fewer authors and more Kesting authors per paper, so under fractional counting each Kesting author carries a larger share. Kesting therefore has 14 papers in total but 12.0 fractional, against Rostval's 4.0.

Why: Covers Kesting's larger shares via fewer authors, and implies Rostval's one-author-per-paper share, but it doesn't clearly say Rostval's small share per multi-author work explains the swap; it does say four-author papers with one Rostval author, so it mostly meets both points, though the explanation is framed mainly around Kesting.

🟡 R3 csv+padding run 3 graded by llm

Full counting gives each work to every affiliated institution, so Rostval's many multi-author works count fully for it. Fractional counting splits each work across its author rows, and Rostval holds about one quarter of most of its works. Kesting's works more often have most or all of their authors at Kesting, so its fractional share rises above Rostval's.

JSON: {"explanation": "Full counting credits each work to every affiliated institution; fractional counting splits each work across its author rows, and Rostval holds about a quarter of most of its works."}

Why: Gets Rostval's small share per multi-author work (one quarter), and says Kesting's works have most or all authors at Kesting, but this is stated as 'more often' rather than as larger shares from solo/few co-authors; largely meets both points, though Rostval's 'only one from Rostval' is only implied by 'about one quarter'.

❌ R3 csv+padding run 4 graded by llm

The premise doesn't hold: there is no swap. Kesting is ahead under both counting methods (13 vs 11 full; 11.5 vs 2.75 fractional).

Why: Rejects the premise and denies a swap; mentions neither Rostval's multi-author shares nor Kesting's larger shares.

❌ R3 csv+padding run 5 graded by llm

The premise is false. In 2024 Kesting (I07) leads Rostval (I09) under both counting methods (full 13 vs 10; fractional 11.5 vs 2.5), so the two don't swap.

Why: Rejects the premise as false and never gives either checklist point (Rostval's small shares of multi-author works, Kesting's larger shares).

❌ R4 csv+markdown run 1 graded by llm

(no answer found)

Why: No answer was given, so neither checklist point is met

❌ R4 csv+markdown run 2 graded by llm

(no answer found)

Why: No answer given; meets none of the checklist points

🟡 R4 csv+markdown run 3 graded by llm

Under full counting, Rostval's 14 works outnumber Kesting's 13. Fractional counting reverses this, because each Rostval work has four authors with one Rostval author each (0.25 per work), while Kesting's works have a larger share of their authors from Kesting.

Why: Says Rostval gets 0.25 per work from four authors (one Rostval), but only vaguely says Kesting has a larger share, without noting many sole-Kesting or few-co-author works

❌ R4 csv+markdown run 4 graded by llm

(no answer found)

Why: No answer was given, so neither checklist point is met.

✅ R4 csv+markdown run 5 graded by llm

Full counting gives each work to every credited institution, so Rostval's 2024 works (mostly 4-author papers) lead 13 to 12. Fractional counting gives each author a share: Rostval's authors take only 0.25 per work (3.25 total), while Kesting's works are mostly Kesting-authored (10.5 total).

❌ R5 csv+ontology run 1 graded by llm

(no answer found)

Why: No answer given; meets none of the checklist points

❌ R5 csv+ontology run 2 graded by llm

The premise does not hold. Kesting (I07) leads Rostval (I09) under both full counting (14 vs 7) and fractional counting (12 vs 1.75), so there is no swap.

Why: Rejects the premise and says there is no swap; meets neither point about Rostval's shared authorship or Kesting's larger shares.

❌ R5 csv+ontology run 3 graded by llm

The premise doesn't hold in my tally. Kesting leads under both methods: 13 vs 12 full, ≈11.2 vs 3.0 fractional. Rostval's 2024 count is the least certain, so recheck it before relying on this.

Why: Rejects the premise and gives no explanation of Rostval's small shares or Kesting's larger shares of 2024 works

❌ R5 csv+ontology run 4 graded by llm

(no answer found)

Why: No answer given; meets none of the checklist points

❌ R5 csv+ontology run 5 graded by llm

The premise is wrong. Kesting does not fall behind under either method; it leads under both (13 vs 11 full, and about 11 vs 2.75 fractional).

Why: Rejects the premise and doesn't give either checklist point (Rostval's shared multi-author works, Kesting's larger shares).

❌ R10 executed run 1 graded by llm

They do not swap in this data, so the premise does not hold. Full: I07 13 vs I09 11. Fractional: I07 11 vs I09 2.75, so I07 leads on both.

Why: Rejects the premise instead of giving the checklist's explanation (Rostval's small shares of multi-author works; Kesting's larger shares); neither point is made.

🟡 R10 executed run 2 graded by llm

The swap happens only under direct counting (Q13): Rostval leads on full counts and Kesting on fractional. Rostval's 11 works each have four authors with one Rostval credit (0.25 each), while Kesting's credited authors sit on fewer-author works (e.g., W22, W528, W595). If units are included (Q12), Kesting leads both ways (13 vs 12 full; 11 vs 3 fractional), so the swap depends on the unit rule.

Why: Gives Rostval's four-author works with one credit (small share) but does not say Kesting holds larger shares via solo or few-author works in general terms beyond examples; mostly meets point 1, point 2 only loosely via 'fewer-author works'

✅ R10 executed run 3 graded by llm

Under direct counting (no units), Rostval has 10 full works, each with four authors and one I09 credit, so 2.5 fractional. Kesting has 9 full works, including single-author W22 and W528, so 6.25 fractional. Full counting favours Rostval on volume, and fractional counting favours Kesting on author share. Including units reverses this: Kesting has 12 full and 10.5 fractional against Rostval's 11 and 2.75, so the swap disappears.

❌ R10 executed run 4 graded by llm

No swap occurs on my counts. Full counts tie at 13 and fractional counts favour Kesting (11 vs 3.25), so the premise does not hold.

Why: Rejects the premise and gives neither the Rostval multi-author small-share explanation nor the Kesting larger-share explanation.

❌ R10 executed run 5 graded by llm

There is no swap. Kesting leads on both counts in 2024: 12 vs 11 full, and 10.5 vs 2.75 fractional (see Q13). The question's premise is false.

Why: Rejects the premise instead of giving either checklist point: Rostval's multi-author works with small shares, or Kesting's larger shares.