Machine peer review

The question

A panel is worth more than a single judge only in so far as its members err independently. Three reviewers who make the same mistake for the same reason are, for the purpose of catching that mistake, one reviewer. Models trained upon much the same literature may well share their blind spots, and the suspicion has a name among those who discuss these matters: the hive mind.

The question is therefore not whether a model can review a manuscript, which the published record allows any reader to judge, but how much a second and a third opinion add to the first.

What the rule amounts to

Under the Society’s rule of decision two models review each manuscript, and a third is called only when they disagree. Counting a recommendation of minor revisions with acceptance, this is exactly a majority of three in which the third vote is sought only where it could alter the result: if the first two agree, the majority is already settled.

An illustration

Suppose each reviewer, taken alone, reaches the wrong decision with probability p. If the three err independently, the majority is wrong when at least two are wrong, which happens with probability

3p²(1 − p) + p³ = 3p² − 2p³.

For p = 0.10 this is 0.028: the panel errs about once in thirty-six decisions where a single reviewer errs once in ten.

Now suppose that with probability ρ the three are moved by a common cause and stand or fall together, and otherwise err independently. This is one simple way of representing shared blind spots. The probability that the majority is wrong becomes

ρ p + (1 − ρ)(3p² − 2p³).

With p = 0.10 and ρ = 0.5 this is 0.064, more than twice the independent figure; and as ρ approaches one the panel is no better than its single member. The advantage of a panel is thus governed less by the skill of its members than by the independence of their errors. These figures are an illustration of the argument and not a finding about any model.

Method

The programme proceeds on two lines. The first is simulation: panels of reviewers are generated by the Monte Carlo method under varying assumptions about the accuracy of each member and the correlation between them, and the reliability of the rule of decision is computed in each case.

The second is the record. Every review on which the journal has acted is published beside its article: 73 reviews of 36 articles to date. The record shows where reviewers agreed, where they differed, and on what grounds, and it is open to any reader who wishes to test the Society’s conclusions or to draw others.

The present stage

This is a standing programme, and it has no end in view while the method remains in use. Its findings will be reported on this page and, where they amount to a finished argument, in the journal. Until then nothing is claimed here beyond the reasoning set out above.