Q47, run 2026-11-04-2053-sonnet

โ† back to the run

Why do Kesting University and Rostval Technical University swap places in 2024 between full and fractional counting?

Tests: Counting method: why the ranking flips.

Expected: Rostval is on four 2024 works (W21, W27, W28, W29), but on each it has one author of four: 4 x 1/4 = 1. Kesting is on three, with larger shares: W22 alone, half of W20, two of the four authors of W21: 2.

๐ŸŸก R1 csv run 1 graded by llm

Full counting gives Rostval 4 works against Kesting's 3. Under fractional counting Rostval's four works (W21, W27, W28, W29) each have only one Rostval author out of four, giving 1.0. Kesting's W22 is single-authored (1.0), and W20 and W21 each give it half a share, giving 2.0.

Why: Gets Kesting's shares (W22 single-author, half of W20 and W21) but says W21 is in Rostval's four works and doesn't state each has four authors with a quarter share; it gives 1.0 rather than a quarter each and doesn't explicitly say they swap places.

๐ŸŸก R1 csv run 2 graded by llm

I09 has more 2024 works (4 against 3), but it is one of four authors on each. I07 has fewer works but larger shares, with sole authorship of W22 and half of W20 and W21.

Why: Gets Kesting's larger shares (sole W22, half of W20/W21) but omits that each Rostval work has four authors, one from Rostval, so a quarter each; only roughly implies it.

๐ŸŸก R1 csv run 3 graded by llm

I09 has more works in 2024 (4 against 3). They are shared among 4 authors, so each gives I09 only 0.25 (1.0 in total). Kesting's W20 and W21 give it 0.5 each and W22 gives it 1, for 2.0.

Why: Gives Kesting's shares (0.5, 0.5, 1) correctly, and Rostval's quarter-shares with four authors, but doesn't say only one author per work is from Rostval, and uses I09 instead of naming the institution; the Rostval point is mostly met, Kesting's W22 single-author review not identified as such.

๐ŸŸก R1 csv run 4 graded by llm

Rostval has more works in 2024 (4 against 3), but each is a four-author paper where it holds only 0.25. Kesting has fewer works but larger shares (W22 is all Kesting, and W20 and W21 give 0.5 each).

Why: Gets Rostval's 0.25 shares across four-author works and Kesting's larger shares (W22 whole, 0.5 on W20/W21), but never states that Rostval has one author on each of its four works, which is implied only loosely.

๐ŸŸก R1 csv run 5 graded by llm

Kesting has fewer 2024 works (3 against 4) but holds larger authorship shares in them. Rostval is only one of four authors on W21, W27, W28 and W29, so it scores 1.0 against Kesting's 2.0.

Why: Rostval's quarter-share per work is covered (1.0 total), but Kesting's shares are not specified (W22 single-author review, half of W20 and W21); it only gives a total of 2.0 and mentions W21 as Rostval's, not clearly identifying Kesting's works.

๐ŸŸก R10 executed run 1 graded by llm

Under full counting, I09 has 4 works (W21, W27, W28, W29) against 3 for I07, so I09 leads. Under fractional counting, I07 gets 0.5, 0.5 and 1 from W20, W21 and W22. I09 gets only 0.25 from each of its four co-authored works, so I07 leads.

Why: Says Rostval's four works give each 0.25 (first point), but doesn't mention four authors with one from Rostval, and Kesting's shares are given only as 0.5/0.5/1 without identifying W22 as a single-author review; roughly meets the shares point, though the full-counting framing differs

โœ… R10 executed run 2 graded by llm

Under full counting, I09 leads 4 to 3, because each work counts once. Under fractional counting, I07 leads 2.0 to 1.0, because I09's four works (W21, W27, W28, W29) each have four authors at different institutions, giving only 0.25 each. I07 holds the sole-authored W22 and half shares of W20 and W21.

๐ŸŸก R10 executed run 3 graded by llm

The premise is slightly off, because under full counting they tie at 4 each in 2024. Kesting leads under fractional counting (3.0 against 1.0) because W22 and W24 are sole-author Kesting works. Each of Rostval's four works (W21, W27, W28, W29) has only one Rostval author among four, so each is worth 0.25.

Why: Correctly gives Rostval's four works at 0.25 each, but credits Kesting with sole-author W22 and W24 rather than W22 plus half of W20 and W21, and denies the swap by claiming a tie under full counting.

๐ŸŸก R10 executed run 4 graded by llm

Under full counting, I09 has 4 works to I07's 3. Under fractional counting, I09 holds only one of four authors on each of its works (1.0 in total), while I07 has the sole-authored W22 and half of W20 and W21 (2.0).

Why: Gets Kesting's shares (W22 sole, half of W20/W21) and Rostval's one-of-four authors per work, but never says each Rostval work is a quarter share; also the identifiers I09/I07 are not named as the institutions.

๐ŸŸก R10 executed run 5 graded by llm

Under full counting every credited work counts once, so I09 leads with 4 works against 3. Under fractional counting I07 leads, 2.0 against 1.0, because it is the sole author of W22 and holds half of W20 and W21. I09 holds only a quarter of each of its four multi-author works.

Why: Gives Kesting's larger shares (sole author of W22, half of W20/W21) but says Rostval holds a quarter of each of its four works without noting each has four authors with one from Rostval; mostly meets point 2, point 1 only loosely