Q47, run 2026-11-05-1234-sonnet

← back to the run

Why do Kesting University and Rostval Technical University swap places in 2024 between full and fractional counting?

Tests: Counting method: why the ranking flips.

Expected: Rostval is on four 2024 works (W21, W27, W28, W29), but on each it has one author of four: 4 x 1/4 = 1. Kesting is on three, with larger shares: W22 alone, half of W20, two of the four authors of W21: 2.

🟑 R1 csv run 1 graded by llm

Rostval has more works in 2024 (4 vs 3), but it is one of four authors on each, so each gives it 0.25. Kesting has a sole-authored W22 and half shares of W20 and W21, so it leads on fractional counting.

Why: Gives Kesting's W22 and half shares of W20/W21, but for Rostval says only 'one of four authors on each' giving 0.25, which is essentially the quarter-share point; however it never explicitly states that only one author per work is from Rostval, so it is nearly but not fully matched

βœ… R1 csv run 2 graded by llm

Rostval has more works in 2024 (4 vs 3), but each is one of four authors, so it gets only 0.25 per work (1.0 in total). Kesting has the sole-authored W22 and half of W20 and W21, giving 2.0.

🟑 R1 csv run 3 graded by llm

I09 has more works (4 against 3) but they are all four-author papers, so each gives only 0.25 (1.0 in total). Kesting's W22 is single-author and its W20 and W21 give 0.5 each, for 2.0 in total.

Why: Gives Kesting's shares (W22 single-author, 0.5 each from W20/W21) and the 0.25 per four-author paper, but doesn't say only one author per Rostval paper is from Rostval; it also doesn't state the swap explicitly.

🟑 R1 csv run 4 graded by llm

Rostval (I09) has more works in 2024 (4 against 3) but only a quarter share of each, because those works have several authors from different institutions. Kesting's works are less shared (W22 is single-author), so its fractional total is 2.0 against Rostval's 1.0.

Why: Gets Rostval's quarter shares from multiple authors and Kesting's single-author W22, but omits that Kesting holds half of W20 and W21 and that each Rostval work has four authors with only one from Rostval.

🟑 R1 csv run 5 graded by llm

Kesting's 2024 works are mostly sole-authored, so each keeps nearly its whole credit. Rostval's four 2024 works (W21, W27, W28, W29) are multi-author, each giving I09 only 1 of 4 shares. Rostval leads on full counting and Kesting on fractional counting.

Why: Gets the Rostval quarter-share point (1 of 4 each) but doesn't cite Kesting's specific larger shares (single-author W22, half of W20 and W21); says only 'mostly sole-authored', which is vague and conflicts with the half shares.

🟑 R2 rdf run 1 graded by llm

Rostval has four 2024 works, each with four authors from four institutions, so each earns only 0.25. Kesting has three works but is the sole or main contributor on them (W22 gives 1, W20 and W21 give 0.5), so it wins on fractional counting.

Why: Rostval part matches (four authors, 0.25 each); Kesting is described as sole/main contributor with W22=1, W20 and W21=0.5, which matches the checklist's larger shares, but it never says W22 is a single-author review, and it says 'three works' unsupported. Mostly covers both points.

🟑 R2 rdf run 2 graded by llm

Kesting has fewer 2024 works than Rostval (3 against 4) but holds larger shares of them. W22 is sole-authored and I07 holds 2 of 4 authors on W21 and 1 of 2 on W20. Rostval is one of four authors on each of its four works, so it gets ΒΌ each.

Why: Gets Rostval's four-author quarter shares and Kesting's larger shares (W22 sole, half of W20/W21), but the answer doesn't state the swap explicitly or the full-vs-fractional ranking; checklist points mostly met except W22 described as 'sole-authored' without calling it a review, and half of W20 and W21 is misstated (W21 2 of 4 = half, W20 1 of 2 = half, fine).

🟑 R2 rdf run 3 graded by llm

I07 has fewer works but larger author shares, because W22 is single-author and W20 is split two ways. I09 is one of four authors on each of its four works.

Why: Captures Kesting's larger shares (single-author W22, split W20) and Rostval being one of several authors, but omits the specifics: four authors per work, a quarter each, and half of W20 and W21.

🟑 R2 rdf run 4 graded by llm

Rostval appears on more works (4 against 3) but only as one of four authors on each, giving 1.0 fractionally. Kesting has the sole-authored W22 and half of W20, giving 2.0.

Why: Gets Rostval's one-of-four-authors share (though says 1.0 total, implying a quarter each) and Kesting's sole-authored W22 plus half of W20, but omits half of W21 and doesn't state the swap explicitly.

🟑 R2 rdf run 5 graded by llm

Counting works favours Rostval, which is on 4 multi-author works. Counting authorship shares favours Kesting. Kesting holds all of W22, half of W20 and half of W21 (2.0), while Rostval holds a quarter of each of its four works (1.0).

Why: Gives Kesting's shares (W22 plus halves of W20/W21) and Rostval's quarter shares, but does not state that each Rostval work has four authors with only one from Rostval.

βœ… R6 rdf+ontology run 1 graded by llm

Under full counting, I09 has 4 works to Kesting's 3. Under fractional counting, Rostval holds only a quarter of each of its four 4-author works (1.0 in total), while Kesting holds half of W20 and W21 and all of W22 (2.0).

βœ… R6 rdf+ontology run 2 graded by llm

Rostval has more works (4 against 3), but each is a four-author work where it holds one author's share of 0.25. Kesting holds half of W20 and W21 and all of W22, so it has 2.0 against Rostval's 1.0.

βœ… R6 rdf+ontology run 3 graded by llm

Counting in full, Rostval has 4 works to Kesting's 3. Counting fractionally, Rostval's four are co-authored four ways and earn it 0.25 each, 1.0 in all. Kesting's include a single-author work and a half share of two others, which earns 2.0.

βœ… R6 rdf+ontology run 4 graded by llm

Under full counting, I09 has 4 works in 2024 against 3 for I07. Under fractional counting, I09 holds only 1 of 4 author slots on each of its works (1.0 in total), while I07 holds W22 whole plus half of W20 and W21 (2.0).

🟑 R6 rdf+ontology run 5 graded by llm

Under full counting I09 has 4 works to I07's 3. Under fractional counting W21 and the other I09 works are split among four authors, giving I09 only 0.25 each. I07 gets full credit for the single-author W22 and half-credit shares of W20 and W21.

Why: Gets the quarter-share for I09 works and Kesting's single-author W22 plus half of W20/W21, but says 'W21 and the other I09 works' without clearly stating each of Rostval's four 2024 works has four authors with only one from Rostval; also doesn't explicitly say the swap in ranking (full vs fractional) is explained by those larger shares, though implied.

βœ… R5 csv+ontology run 1 graded by llm

Under full counting I09 has 4 works to I07's 3. Under fractional counting I07 leads 2.0 to 1.0, because I09's works (W21, W27, W28, W29) each have four authors from several institutions, so I09 gets only 0.25 of each. I07 has two single-institution works (W20 at 0.5 and W22 at 1) plus a half share of W21.

🟑 R5 csv+ontology run 2 graded by llm

Full counting gives each institution a whole work, so Rostval leads with 4 against 3. Fractionally, Kesting leads because W22 is solely its own and W20 is half its own. Rostval has only one author's quarter share in each of its four multi-author works.

Why: Gives Kesting's larger shares (W22 sole, half of W20) but omits W21 and does not state that each Rostval work has four authors, with one from Rostval, as a quarter each, though it mentions one author's quarter share in multi-author works.

βœ… R5 csv+ontology run 3 graded by llm

In 2024 Rostval has more works under full counting (4 against 3). Its works (W21, W27, W28, W29) are four-author collaborations in which it holds one author, so it gets only 0.25 of each work. Kesting's W22 is sole-authored and W20 and W21 give it half or more, so it gets 2.0 against 1.0 under fractional counting.

🟑 R5 csv+ontology run 4 graded by llm

I09 has more works in 2024 (4 against 3), but they are all multi-author, multi-country papers worth 0.25 each (1.0 in total). I07's W22 is sole-authored and W20 and W21 give it half shares, so it totals 2.0.

Why: Gives Rostval's 0.25 shares (multi-author) and Kesting's sole-author W22 plus half shares of W20/W21, but doesn't state four authors with only one from Rostval per work; also doesn't mention the swap explicitly

βœ… R5 csv+ontology run 5 graded by llm

Rostval has more works in full counting but a smaller share in fractional counting. Each of Rostval's four 2024 works (W21, W27, W28, W29) has four authors with only one from I09, so it earns 0.25 each, 1.0 in total. Kesting's W22 is single-authored (1) and W20 and W21 give it 0.5 each, so 2.0 in total.

βœ… R4 csv+markdown run 1 graded by llm

I09's 4 works each have 4 authors from different institutions, so it earns only 0.25 per work. I07 has the sole-authored W22 plus half shares of W20 and W21, giving 2.0 against 1.0 for I09. The full-count lead of I09 (4 against 3) therefore reverses under fractional counting.

🟑 R4 csv+markdown run 2 graded by llm

Under full counting, I09 has 4 works in 2024 (W21, W27, W28, W29) against 3 for I07. Under fractional counting, I09 gets only 0.25 of each, since its authors are 1 of 4 on each work, giving 1.0. I07 gets 2.0 because it has the sole author on W22 and half of W20 and W21.

Why: Explains Rostval's quarter shares (1.0 total) and Kesting's single-author W22 plus half of W20/W21, but doesn't state that all four Rostval works have four authors each; it assumes it from the totals and omits the explicit swap framing.

🟑 R4 csv+markdown run 3 graded by llm

Rostval's four 2024 works (W21, W27, W28, W29) each have one Rostval author among four, so they count 4 in full but only 1.0 fractionally. Kesting's 3 works carry a larger share of their authors, giving 2.0.

Why: Rostval's quarter-share reasoning is correct, but Kesting is only said to have a larger share, with no mention of the single-author W22 or the half shares of W20 and W21.

βœ… R4 csv+markdown run 4 graded by llm

Rostval has more works (4 against 3) but each has only one of four authors from I09, so its share is 1.0. Kesting's W22 is single-authored and W20 and W21 give half-shares, so it gets 2.0.

βœ… R4 csv+markdown run 5 graded by llm

I09 has more works in 2024 (4 against 3) because it appears on W21, W27, W28 and W29. In each of those it is only one of four authors' institutions, so its fractional total is 1.0. Kesting's W22 is single-author and its shares on W20 and W21 are larger, giving 2.0.

βœ… R3 csv+padding run 1 graded by llm

Rostval has more works in 2024 (4 vs 3), but each is a four-author paper with one Rostval author, which gives 0.25 each (1.0 total). Kesting has the sole-authored W22 (1) plus half shares of W20 and W21 (2.0 total), so it leads on fractional counting.

🟑 R3 csv+padding run 2 graded by llm

Rostval has more 2024 works (4 against 3), but each is a four-author paper where it holds only 0.25. Kesting's W22 is a sole-author paper worth a full 1, so its fractional total is higher.

Why: Correctly gives Kesting's sole-author W22 as the larger share, but says nothing of Kesting's half shares in W20 and W21, and only implies Rostval's quarter shares (four authors, one from Rostval) without saying that each of its four works is shared that way.

🟑 R3 csv+padding run 3 graded by llm

Rostval has more 2024 works (4 against 3) but is only one of four authors on each. Kesting has the sole-authored W22 and half of W20 and W21, so it leads 2.0 to 1.0 once each work is split among its authors.

Why: Kesting's shares (W22 sole, half of W20 and W21) are correct, but the answer never says each Rostval work has four authors and a quarter each; it only says 'one of four'.

🟑 R3 csv+padding run 4 graded by llm

Under full counting, Rostval has 4 works against Kesting's 3. Under fractional counting, each Rostval work has four authors with only one at Rostval (4 Γ— 0.25 = 1.0). Kesting's W22 is single-authored, so it scores 2.0.

Why: Gives Rostval's quarter-share reasoning, but for Kesting only says W22 is single-authored and omits the half shares of W20 and W21.

🟑 R3 csv+padding run 5 graded by llm

Rostval's 2024 works each have 4 authors with only one at Rostval, so each adds just 0.25 (1.0 in total). Kesting's three works have fewer co-authors and concentrate its authors, so it gets 2.0.

Why: Gets Rostval's quarter-share per work (four authors, one from Rostval), but doesn't identify Kesting's single-author review W22 and half shares of W20 and W21.

❌ R10 executed run 1 graded by llm

They don't swap. In 2024 both have 4 works under full counting (tie), and under fractional counting Kesting leads 3.0 to 1.0, because Rostval is one of four co-institutions on each of its works.

Why: Denies the swap and says full counting is a tie; it also never states the quarter-share reasoning for Rostval's four works or Kesting's W22 and half of W20/W21 shares.

🟑 R10 executed run 2 graded by llm

Rostval has more works under full counting (4 against 3) because W27, W28 and W29 each have I09 credited. Those works have four authors from different institutions, so Rostval gets only 0.25 each, giving 1.0 in total. Kesting's solo-authored W22 gives it a full 1, and it totals 2.0.

Why: Gets Rostval's quarter shares (four authors, 0.25 each) but only mentions Kesting's solo W22, omitting its half shares of W20 and W21; its totals also don't match the checklist's Kesting shares.

❌ R10 executed run 3 graded by llm

They don't swap. In 2024 they tie at 4 under full counting, and Kesting leads under fractional counting, 3.0 to 1.0. Kesting's works have fewer co-authors from other institutions, so fractional counting keeps more of each work.

Why: Denies a swap and gives a tie/different explanation; misses the Rostval quarter shares and Kesting's W22 and half of W20/W21 shares.

🟑 R10 executed run 4 graded by llm

Under full counting, I09 has 4 works against I07's 3. Under fractional counting, I09 gets only 0.25 from each of its four works because they have four authors each (1.0 in total). I07 gets 2.0, mainly from W22, which is single-author (1) and W20 and W21 (0.5 each).

Why: Gives Rostval's four-author quarter shares and Kesting's W22/W20/W21 shares, but never names the two universities as the swap, and does not state that only one of the four authors is from Rostval; the one-from-Rostval point is only implied by the 0.25 share.

❌ R10 executed run 5 graded by llm

They do not swap. In 2024 they tie under full counting (4 each) and Kesting leads under fractional counting (3.0 against 1.0), so the question's premise doesn't hold.

Why: Denies the swap and never gives the checklist reasons (Rostval's four-author works at a quarter share each; Kesting's W22 and half of W20/W21).