A few months ago, I received an email that asked me to serve on the ICWSM 2014 Program Committee. The role of PC member for this conference entails reading several submitted papers to the conference and then writing up reviews that would then be used by the senior members to come to a decision about accepting or rejecting the papers. I was really excited to be asked to be a part of this process because it was my first time seeing what goes on behind the scenes of how papers get vetted, reviewed, and deliberated on by the research community. Unfortunately I could find few resources for how to write a proper review and did not come across any tailored to this specific research community (or even the general research communities of HCI, CSCW, or social computing).
Nevertheless, with the help of my research advisor who provided several useful tips, I came away from the process with a better understanding of how to write a good review. I also found reading papers from the point-of-view of a reviewer eye-opening and informative for myself as a researcher and writer. As a result, I am writing down some of my thoughts and findings in the hopes that they may be useful for other beginning reviewers going through the same experience. Please note that this is not meant to be a comprehensive how-to for writing reviews – only an account of the thought-process, revelations, and observations of a 1st time reviewer!
My initial reaction to being invited to review was excitement, followed closely by anxiousness. I was worried that as a beginning researcher I would not find anything substantial to say about any of the submissions, written as they were by more experienced researchers than myself. Surely any of the meager suggestions or recommendations that I could potential prescribe would have been thought of and taken care of already?
Once I actually received my assigned papers, however, I was surprised to find that there was a good amount of variation in the papers in terms of quality of writing and methodology, and incidentally, I could in fact come up with comments for each of the papers (though whether these comments were of any value I couldn’t yet be certain – more on that later).
Some things that I noticed while reading papers from the lens of a reviewer for the first time:
Grammatical errors, imprecise or undefined wording, and poor sentence structure were glaringly noticeable, and I could sense my perception of the paper quickly souring upon encountering more than a few of them. Noticing this, I tried my best to suppress my misgivings and overlook such errors because it was apparent that some of the papers were written by non-native English speakers, and I thought it unfair for them to be penalized.
On a wider note, I realized how incredibly important it is to be a good writer and to think deeply about organization and presentation. As a reviewer, it was almost a joy to read the papers that conveyed ideas clearly, presented well-thought-out graphs and tables, and told a well-structured story that flowed coherently from section to section. On the flip side, it was frustrating to read papers that had sections that seemed misplaced or irrelevant, or graphs and tables where it wasn’t clear what they were trying to express.
There were two papers sections in particular that stood out to me as surprisingly important:
I could sense alarm bells going off when I read a Related Work section and could think of or could quickly Google related research that was not mentioned. It wasn’t necessarily a specific paper, but when a particular research direction or theme that was very relevant to the work of the paper was not given mention, it made me wonder what else the writers missed. As someone who is not the pre-eminent expert on every topic of the papers I read nor in the position of replicating each paper’s work, I couldn’t be certain that everything the writers were asserting was true. So reading a comprehensive Related Work section somehow made me feel more confident in the writers and more inclined to believe their assertions.
The Discussion section I found to also be extremely important. One might think that in order to write a strong research paper, it’s best to emphasize the strengths of the method or highlight only the positive findings and obfuscate, ignore, or minimize bad or inconclusive findings. In fact, I found that the opposite is true. I found myself much more willing to take a paper’s findings and conclusions at face value if the authors openly talked about drawbacks, caveats, and failures they had while doing their work. It was also interesting to see how authors tried to understand or conceptualize their findings, even if it wasn’t rigorous or a main part of their paper. I found it much more interesting as a reader to be presented with findings supported by a plausible explanation or put in context of a larger model rather than being given a slew of numbers at the end with no discussion at all.
After writing my initial reviews, I sent them off to my advisor for him to give a quick look-over to make sure I wasn’t putting my foot in my mouth. He gave me some great tips which I summarize below:
- Start out reviews with a short summary of the paper. It’s useful for reviewers to remember what paper they’re reading about and helps authors know if they’re getting their main ideas across properly. Don’t forget to write within the summary what aspect of the paper is novel. This forces the reviewer to focus on the novelty of the work, which may affect the rating given to the paper. I recall cases where I actually struggled to put into words the novelty of a paper and as a result, realized I should bumped my ratings down.
- Be careful of subjective statements that start with phrases such as “I would have liked…” because it’s unclear to the meta-reviewer and authors whether this is an objective problem with the paper or a subjective preference. The former is a strike against the paper while the latter may be interesting to the authors but shouldn’t count against the paper in terms of acceptance into the conference. When going through my reviews, I noticed that I had included several of these phrases – I guess because I was a first-time reviewer and afraid of coming off too strong or assured in my review, especially if I turned out to be wrong.
After turning in my reviews, I got a chance to see how other reviewers reviewed the same papers. Some reviews brought up points I had overlooked or had diverging opinions from me, which was useful for me to see the other points of view that I didn’t consider as well as the range of perspectives. The best reviews in my opinion were the ones that demonstrated their expertise in that particular niche by backing up their points with citations and including recommendations and papers to look at along with their critiques. I could see how these reviews would be really useful for the authors to refine their work. Finally, it was a validating experience to see when other reviews brought up similar points to my own and when the meta-reviewers cited my review or mentioned points from my review as helpful for forming their final opinion. Here was concrete proof that many of my comments were actually valuable.
So in the end, my worst fears of being wildly off the mark in my reviews never materialized, and I finished the experience more confident in my research instincts and more understanding of the mentality of paper reviewers and thus how to frame, organize, and style my research writing. I think being part of the review process will help me become a better writer and ultimately, a better researcher and active member of the research community, and I encourage program chairs and organizers of conferences to invite PhD students and newer members of the community to participate in the review process.
* This post is cross-posted at the Haystack Group blog.
* While I’ve learned a great deal from this process, there are many, many veteran reviewers out there who have many more tips that they’ve cultivated or learned over time. Therefore, I’m keeping a list of comments to this piece and further advice I receive from others here:
– “…always talk about the weaknesses and limitations of your approach in a paper, but don’t have that be the last thing you talk about. I remember once ending a paper with the “Limitations” section right before a short conclusion, and my MSR boss telling me how that leaves a negative last impression.”
– My advisor also mentioned that a great (and hilarious) resource for reviewers is “How NOT to review a paper: The tools and techniques of the adversarial reviewer” by Graham Cormode, which also has citations to other useful guides on reviewing.
– This guide written for CHI 2005 is several years old but is a really thorough look into how to write a proper review, including useful examples of both suitable and unsuitable reviews.