Rewiewing is a Contract
Tags: academia, musings
It is a beautiful day outside, but I do not feel my usual chipper self. Huddled in my robe, juiced up on some tea, I nevertheless start doing my third emergency review for one of the largest machine learning conferences. And I am not the only one. A good friend and colleague of mine is an area chair of this conference; he sounds defeated when he mentions that even as we are way past the deadline for submitting reviews, there will be at least a couple of papers that will not receive more than two reviews. Despite what you think about peer review, this makes an already-tenuous situation even more random. How did we arrive at this point?
Some perspective: at last-year’s LoG, the first ‘Learning on Graphs’ conference, I was honoured to be a programme chair, and I remember almost frantically searching for emergency reviewers at short notice because the assigned reviewers were ghosting me—no reaction, nothing. Kyunghyun Cho, who is heroically in charge of multiple machine learning conferences now, aims to encourage people by tweeting about the progress of this year’s reviewing. At NeurIPS 2022, he creatively tried to cajole people into reviewing, documenting his own descent into the abyss while chasing reviewers.
As for me, I stared the abyss down, and, uncharacteristically for me, I feel anger at reviewers flaking, because reviewing is contract. If you agree to review something, it is your responsibility to ensure that you are able to do so. No one forces you to review something. You agree to provide a service for the community, and a crucial one at that. In fact, reviewing is the foundation of our system of disseminating research. There are many ways to be bad at reviewing, but the worst one is to never show up despite volunteering for that.
Now, I fully get that you might be unable to do your review—maybe you find that the papers are more complicated than anticipated, or something else crops up in your life that demands your immediate attention. These things happen, and it is absolutely right to prioritise your own well-being or your time. The only thing anyone in a position to oversee conference reviewing is asking for is to communicate any problems to your area chair or the program chairs, if need be. With OpenReview, I can easily un-assign you as a reviewer, no harm done. However, it is morally unacceptable to sign up for the reviewing process—voluntarily—and then never communicate, not even in the earlier phases of reviewing. The silence we get from reviewers that do their job is homeomorphic to the silence of reviewers unable or unwilling to do their job, so please, please, please reach out to the conference organisers early on. A simple e-mail like this will do the trick:
Dear AC,
I unfortunately will not be able to review [all|some] of my papers assigned to me. Please un-assign them from my profile.
[NAME]
No one will think any less of you if you do this. But apparently even such a brief interaction is not working for many reviewers, and they just stay silent. Please do not be like this. The community depends on you. I could appeal to a moral imperative, or to the tragedy of the commons that awaits us if we all stopped reviewing for conferences but expect our own papers to be reviewed. I am not sure whether such appeals are ultimately useful, but I urge all of you to think about whether you are an active scientific participant in our community or not. Do not deprive the community of your voice and opinions. Write the reviews that you would like to receive and please be virtuous when you encounter an insurmountable obstacle preventing you from reviewing.
Creating a great community requires everyone to pitch in, but in the end, the result will be amazing and beneficial for all.