Journals should be experimenting with LLMs in their editorial workflow

I recently wrote a post about using Claude to peer review an academic paper, and the decent job it did. Based on that experience, I started thinking about the probable impact on journal editorial workflows, a significant part of which is the peer review process.

If I was still on an editorial board of a journal with a budget, this is what I’d be setting up now:

• A randomised experiment, where some proportion of all submissions are automatically copied into a parallel process.
• Run the peer review as normal and gather the submissions from both reviewers.
• Once the submissions have been reviewed by the responsible editor, that editor ‘releases’ the copied version of the article into a third peer review that’s conducted by a language model.
• A separate panel of editors reviews the 3 reviews and compares the recommendations.

Over time, as the editors see a significant overlap in the outputs of the LLM, I imagine a reversal of workload, where the bulk of submissions go via the parallel process (i.e. they’re reviewed by the LLM first). After this, a randomised selection of those LLM-generated reviews are sent out to peer reviewers for checking, and they check the reliability of the review.

As our trust in LLMs grows, I don’t see any reason why we’d even bother sending them out for checking. You’ll write your article, submit to the journal, and get an outcome immediately. As the system becomes more sophisticated, Revision and Major Revision decisions will be accompanied by recommendations for changes, and you’ll get that feedback in seconds.

I also believe that the quality of the reviews will be better. It’ll have less personal bias, and will be more collegial because editors can ask the LLM to adopt certain tones (for example, you could ask your peer review LLM to take on a nurturing and supportive tone). I think the LLM-generated reviews can be guided to provide more useful, supportive, and constructive feedback to authors.

I may be wrong about the details, but even with no further changes in LLM technology, and no fine-tuning of the models, I don’t see any reason why we shouldn’t be doing this already. This would reduce the significant amount of time spent on the peer review process and, in my opinion, lead to more collegial, trustworthy, and constructive peer reviews.