Q47, run 2026-11-04-2010-sonnet

← back to the run

Why do Kesting University and Rostval Technical University swap places in 2024 between full and fractional counting?

Tests: Counting method: why the ranking flips.

Expected: Rostval is on four 2024 works (W21, W27, W28, W29), but on each it has one author of four: 4 x 1/4 = 1. Kesting is on three, with larger shares: W22 alone, half of W20, two of the four authors of W21: 2.

🟑 R1 csv run 1 graded by llm

Rostval has more works in full counting (4 vs 3). In each of W21, W27, W28 and W29 it is only one of four authors, so it earns 0.25 per work (1.0 total). Kesting has W22 alone (1) plus half of W20 and W21, giving 2.0.

Why: Correctly gives Rostval's four works with one of four authors (0.25 each) and Kesting's W22 plus halves of W20 and W21, but it also says W21 is one of Rostval's works, which conflicts with the stated Kesting share, and it does not say Kesting's shares are larger.

🟑 R1 csv run 2 graded by llm

Fractional counting splits shared works. Kesting's W20 and W21 are each half shared with other institutions, giving 2.0 against Rostval's 4 works at a quarter-share each (1.0). Under full counting Rostval's 4 beats Kesting's 3.

Why: Correctly notes Rostval's works are quarter-shares and Kesting's W20/W21 are half-shares, but omits that each Rostval work has four authors with one from Rostval, and omits Kesting's single-author review W22; its 3-work full count is also not tied to W22.

🟑 R1 csv run 3 graded by llm

Rostval appears on more works (4 vs 3 in 2024), but on each it is one of four co-authors (0.25 share each). Kesting has the sole-author W22 (1) and half-shares of W20 and W21, giving 2.0 vs 1.0.

Why: Gives Rostval's quarter share per work and Kesting's W22 plus half-shares of W20/W21, but says Rostval has 4 works in 2024 vs Kesting 3 only partly matching; the answer states Rostval 4 works, 0.25 each, so it meets both points, though it never explicitly says 'four authors with one from Rostval' as a fixed fact for all four works

βœ… R1 csv run 4 graded by llm

Rostval's four 2024 works each have four authors, so it gets 1/4 of each (1.0 in total). Kesting has the single-author W22 and half of W20 and W21 (2.0 in total).

🟑 R1 csv run 5 graded by llm

Full counting gives one credit per work, so Rostval has 4 against Kesting's 3. Fractional counting weights by author share: Rostval's four 2024 works are all multi-author and give 0.25 each (1.0 total). Kesting has the single-author W22 (1.0) plus half shares of W20 and W21 (2.0 total).

Why: Covers Rostval's quarter shares (0.25 each) and Kesting's W22 plus half of W20/W21, but doesn't state that each work has four authors with one from Rostval; it also doesn't explicitly say the ranking swaps, though that is implied.

🟑 R2 rdf run 1 graded by llm

I09 has more 2024 works (4 against 3) but only a ΒΌ share of each, since they have four authors each. I07 has fewer works but larger shares: W22 has a single author and W20 and W21 are shared half and half.

Why: Gets Rostval's quarter shares (four authors each) and Kesting's larger shares (W22 single author, half of W20/W21), but says 4 works for Rostval vs the key's four works too; matches mostly, yet uses IDs I09/I07 without naming the institutions and omits that W22 is a review

βœ… R2 rdf run 2 graded by llm

Rostval has more works in 2024 (4 vs 3), but it is only one of four authors on each, so each work is worth 1/4. Kesting has fewer works but larger shares: W22 is entirely Kesting, and W20 and W21 are half Kesting. That gives Kesting 2.0 against Rostval's 1.0.

🟑 R2 rdf run 3 graded by llm

The two universities swap because fractional counting divides each work among its authors. Rostval's four 2024 works each have four authors from four different institutions, so it earns only 1.0 in total. Kesting's W22 has a single author, so it earns 2.0 despite having fewer works.

Why: Gets Rostval's quarter shares (1.0 total) and Kesting's single-author W22, but omits Kesting's half shares of W20 and W21 and doesn't state the swap direction explicitly

βœ… R2 rdf run 4 graded by llm

Rostval has more works (4 against 3) because it appears on W21, W27, W28 and W29. In each of those it is only one of four authors, giving 0.25 each and 1.0 in total. Kesting has fewer works but larger shares: W22 is sole-authored (1) and W20 and W21 give it 0.5 each, for 2.0.

🟑 R2 rdf run 5 graded by llm

Rostval has more works in 2024 (4 vs 3), but it is usually one of four authors on its works (ΒΌ each). Kesting's W22 is single-authored and W20 and W21 give it Β½ each, so its fractional total (2.0) beats Rostval's (1.0).

Why: Gets Kesting's larger shares (W22 single-author, half of W20/W21) and the swap, but says Rostval is 'usually' one of four authors rather than stating each of its four works has four authors with one from Rostval.

βœ… R6 rdf+ontology run 1 graded by llm

Rostval (I09) has more works in 2024 (4 against 3), but each is one of four authors, so its share is only 0.25 per work. Kesting (I07) has the sole-authored W22 and half shares of W20 and W21, giving 2.0 against 1.0.

🟑 R6 rdf+ontology run 2 graded by llm

I09 has more 2024 works (4 vs 3), but its works are split among many co-authors from other institutions. Each I09 work earns only 0.25, while I07 holds W22 outright and half of W20 and W21.

Why: Gives Kesting's shares (W22 whole, half of W20/W21) and 0.25 per work for I09, but doesn't state four authors with only one from Rostval; also says 'many co-authors' and omits that Rostval has four works vs 3 only partially matching.

🟑 R6 rdf+ontology run 3 graded by llm

Rostval has more works (4 against 3), but each is shared with several outside authors (W21, W27, W28, W29 give it 1 of 4 authors each), so it earns only 1.0. Kesting's W22 is solely its own and W20 is half, giving 2.0.

Why: Gets Rostval's quarter shares (1 of 4 authors each) and Kesting's W22 sole and W20 half, but omits that Kesting also holds half of W21 and mischaracterizes W21 as a Rostval work; it also lists W21 as Rostval's 4-author work while Kesting shares it.

βœ… R6 rdf+ontology run 4 graded by llm

I09 leads under full counting (4 works against 3) because each of its works counts whole. Under fractional counting I09's works are 4-author works with only one I09 author each (0.25 each), while I07 has the solo-authored W22 and half shares of W20 and W21, so I07 leads (2.0 against 1.0).

βœ… R6 rdf+ontology run 5 graded by llm

Under full counting each work counts once per institution, so Rostval's four 2024 works beat Kesting's three. Under fractional counting, Rostval's four works are shared among four authors each (about 0.25 per work), while Kesting's W22 is sole-authored and W20 and W21 give it 0.5 each. That puts Kesting at 2.0 against Rostval's 1.0.

βœ… R5 csv+ontology run 1 graded by llm

Under full counting Rostval has 4 works to Kesting's 3. Under fractional counting Kesting has 2.0 to Rostval's 1.0. Rostval's four 2024 works (W21, W27, W28, W29) are all four-author papers with one Rostval author each, worth 0.25 apiece. Kesting has the sole-authored W22 (1) plus half of W20 and W21 (0.5 each).

🟑 R5 csv+ontology run 2 graded by llm

In full counting I09 wins with 4 works to 3. In fractional counting I07 wins with 2.0 to 1.0. I09's four works (W21, W27, W28, W29) are each shared by 4 authors, so it gets 0.25 each. I07 has the solo work W22 and gets half of W20 and W21.

Why: Covers Kesting's solo W22 and half shares of W20/W21 and the 0.25 shares for I09, but never names the institutions as Kesting/Rostval, and says W21 is one of I09's works (W21 is also half-shared by I07), which is an inconsistency with the key's four Rostval works; mostly matches though.

βœ… R5 csv+ontology run 3 graded by llm

In 2024, I09 leads under full counting with 4 works against 3. Under fractional counting it falls behind, with 1.0 against 2.0. All four I09 works have four authors, of whom only one is from I09 (0.25 each). I07 has the sole-authored W22 and half of W20, which outweigh its 0.5 from W21.

🟑 R5 csv+ontology run 4 graded by llm

Under full counting Rostval has 4 works against Kesting's 3. Each Rostval work has four authors and only one is from I09, so Rostval gets 4 Γ— 0.25 = 1.0. Kesting's shares (0.5, 0.5, 1) total 2.0.

Why: Correctly gives Rostval's quarter-share per work, but Kesting's shares are given only as (0.5, 0.5, 1) without naming W22 as single-author review and W20/W21 as half-shares.

βœ… R5 csv+ontology run 5 graded by llm

Kesting leads on fractional counts because it holds a full share of W22 and half of W20 and W21. Rostval leads on full counts because it appears on four works (W21, W27, W28, W29), but on each it is only one of four authors, so each work gives it a quarter share.

🟑 R4 csv+markdown run 1 graded by llm

I09's four 2024 works each have four authors with only one from I09, so it gets 1.0 fractionally. I07's W22 is sole-authored and it holds two of the four authors on W21, which gives it 2.0, even though it has fewer works (3 against 4).

Why: The Rostval quarter-share point is correct, but the answer omits half of W20 and credits Kesting's 2.0 to W22 plus W21 alone, which only sums to 1.5.

βœ… R4 csv+markdown run 2 graded by llm

I09 has more works in full counting (4 against 3), but its works (W21, W27, W28, W29) each have four authors from several institutions, so it gets 0.25 per work, 1.0 in total. Kesting has the sole-authored W22 and half-shares of W20 and W21, 2.0 in total, so the order flips.

🟑 R4 csv+markdown run 3 graded by llm

I09 has more works under full counting (4 against 3), but each of its works is shared among four authors from different institutions, so it earns only 0.25 per work (1.0 in total). I07 has W22 as a sole-authored work and half of W20, so it earns 2.0 and leads under fractional counting.

Why: Rostval's quarter-share point is correct, but the answer credits Kesting with only W22 and half of W20 and omits half of W21 (its 2.0 total needs it).

βœ… R4 csv+markdown run 4 graded by llm

In 2024 Rostval has 4 works against Kesting's 3, so it leads under full counting. Its works are shared among 4 authors, so each gives it only 0.25 (total 1.0). Kesting's W22 is single-authored and gives it a full 1, and W20 and W21 give 0.5 each (total 2.0).

🟑 R4 csv+markdown run 5 graded by llm

I09 leads on full counting (4 works against 3) because it is on W21, W27, W28 and W29. On fractional counting I07 leads (2.0 against 1.0) because W22 is solely I07's and W20 and W21 give it half shares. I09's four works are all four-author papers where it holds only one quarter each.

Why: Says I09's four works are four-author papers with one quarter each, and I07 holds W22 solely plus half of W20/W21, but never maps these to Rostval/Kesting by name; both checklist points are essentially covered, though the explanation of 'four 2024 works' is consistent. Strictly, it meets both points in substance but uses IDs only.

🟑 R3 csv+padding run 1 graded by llm

Rostval has more works (4 against 3), but each is shared among four authors, so each earns only 0.25 (1.0 in total). Kesting's W22 is single-authored and W20 and W21 give 0.5 each, for 2.0.

Why: Gives the quarter-share point for Rostval (four authors, 0.25 each) and Kesting's W22 single-author plus 0.5 each from W20/W21, but doesn't say only one of the four authors is from Rostval; implicit in 0.25. Mostly meets both, though it never states the swap itself.

🟑 R3 csv+padding run 2 graded by llm

Rostval has more works in 2024 (4 against 3), but it is only one of four authors on each, so each work earns it 0.25. Kesting's works have fewer authors (W22 has one, and W20 and W21 are split fairly evenly), so it earns 2.0 in total.

Why: Gives Rostval's quarter share per work (0.25 each) but does not say each has four authors with one from Rostval explicitly; Kesting's larger shares are only vaguely described (W22 single-author is right, but W20/W21 are 'split fairly evenly' rather than half each).

βœ… R3 csv+padding run 3 graded by llm

Rostval's 2024 works are four multi-author papers, each giving it only a 0.25 share. Kesting has the sole-authored W22 worth a full 1, plus 0.5 each from W20 and W21.

βœ… R3 csv+padding run 4 graded by llm

The two universities' 2024 works are shared differently between authors. Kesting has 3 works but gets more credit per work (W22 is single-author, and W20 and W21 give it 0.5 each), so it scores 2.0. Rostval has 4 works (W21, W27, W28, W29), but each has 4 authors and Rostval has one of them, giving 0.25 per work and 1.0 in total.

βœ… R3 csv+padding run 5 graded by llm

I09 has more works in 2024 (4 against 3) but a small share of each: 0.25 of four-author papers. I07 has the sole-authored W22 and half of W20 and W21, so it leads on fractional counting (2.0 against 1.0).