How to write a good article review

Describes what a proper review of a journal or conference article in Computer Science or Software Engineering should look like and how to prepare it.

1. Structure

A review should have three sections:

1. A summary,
2. an overall discussion and judgment,
3. and detailed comments.

This section explains what should be in each of these; the next section will explain how to write them (hint: not in this order).

1.1 Summary

Summarize the article in your own words. The summary should be short, typically two or three sentences. Use your own words exclusively, never repeat the phrasing of the article's abstract or body. The summary should not attempt to be comprehensive (like an abstract would), but rather focus only on what you perceived as the essence of the work with respect to your subsequent judgment.

The summary is quite important and serves several purposes:

• It shows the authors that you have understood their work properly.
  They will not take critical comments seriously otherwise. If they don't, most of your review's value is wasted.
• For conferences, it explains your perspective on the article to your co-reviewers.
  Without it, they have a much harder time to take your differing opinion seriously or to understand where it comes from.
• For journals, it explains your perspective on the article to the editor and much increases the insight she can gain from reading the rest of your review (even though she has read the article's abstract before).
• When a journal article comes back for re-review several months later, the summary is your quickstarter for remembering what your own original review was all about.

1.2 Overall judgment

The overall judgment should explain

• What are the 2 to 5 main strengths of this article?
• Why are they strengths? (This is needed only if the authors have failed to explain this properly. It will usually be empty.)
• What are the 2 to 5 main issues with this article?
• Why are they problematic? How problematic? Do they damage any of the strengths? How?
• Overall, what is the key tradeoff between strengths and weaknesses that leads to your recommendation?
  (Often this will simply be one single weakness that is just too strong or a mutually reinforcing pair of weaknesses)
• What is your recommendation: accept or reject?

Having all the strengths and weaknesses as itemized lists and the discussion as a short one-sentence text below is the easy form in which to write this.

Much more appropriate, though, is usually a form that is essentially a short essay that makes its key points in continuous text (and perhaps embodies remaining parts of the lists as lists somewhere).

Any review depends a lot on the particular reviewer, so there is no such thing as an objective review. Therefore, a good review does not attempt to create the illusion of objectivity, but rather takes a certain point of view quite explicitly. This decision on the point of view (and then properly working out its consequences) is the critical aspect of a good overall judgment.

1.3 Detailed comments

The detailed comments are an itemized list of anything you find worth commenting, such as

• typos (only if there are only few of them)
• hard to understand sentences
• bad structuring of sections
• missing information (facts; related works; rationale; discussion of alternatives; threats to validity; etc.)
• ambiguities
• logical or factual inconsistencies
• dubious or risky reasoning
• and whatever else comes to mind while reading

Put them simply in reading order. If the list is long, structure it by repeating the section titles. Provide clear coordinates for the part of the text that each entry relates to. Some people use page and paragraph numbers for this, but quotes (anything from a single unique word up to a pair of sentences) are much more useful:

• easier to read for the recipients,
• less ambiguous,
• they can later be understood without having the original article at hand,
• they can still be found (if still valid) after a journal article has been reworked.

The goal of each and every detailed comment should be to give the authors an opportunity for improving their article.
They are not there to show anybody how diligent or how hard-working the reviewer is or how much sharper, more knowledgeable, or more oh-so-right than the authors.

2. How to create it

1. Read the article thoroughly, start to finish.
   Record as many detailed comments as you find appropriate (don't exaggerate or the effort might kill you).
   Fully spell out each detailed comment as you first record it; a too-short, incomprehensible comment is not helpful.
2. Now think about the article as a whole:
   What is its story? Its focus?
   Maybe use this checklist: http://sciencecareers.sciencemag.org/career_magazine/previous_issues/articles/2001_04_20/nodoi.5045631236818121057
3. Now write the summary
4. Now think about the article as a whole again:
   Given that focus, what are its main strengths? Weaknesses?
5. Write strengths and weaknesses up as two concise separate itemized lists
6. Develop and write up an argument: What are the most important strengths and weaknesses? Why? How do they trade off?
   (To do this, I usually find it helpful to identify 2-5 recurring types of issue, tag the corresponding items from my detailed comments accordingly, and refer to the hash tags in the list of weaknesses or the argument.)
7. Check your review by going through the ACM Empirical Standards checklist that is appropriate for the given work: Have you forgotten anything important? Have you been overly critical somewhere? If so, correct your review.
   https://web.cs.dal.ca/SIGSOFT-Empirical-Standards/
   https://github.com/acmsigsoft/EmpiricalStandards
8. Explicitly formulate your recommendation: Reject/accept.
   For journals, a reject should become a major revision iff the problems are only with the writeup, not with the research itself.

3. Consider signing your review

My reviews all start with the following block of text

This review is provided under a Creative Commons Attribution license (CC-BY).
This means the author must be mentioned when the text is distributed.
The author of this review is Lutz Prechelt.

I feel this is the best quality assurance for my reviewing I can get. When I attach my name to the review, I will make damn sure - that each criticism is well-explained and proper evidence presented, - that I emphasize strengths as well, not only weaknesses (not only in the overall evaluation, but in the detailed comments, too), - that I balance strengths and weaknesses carefully in my recommendation, - that I target any criticism to the work only, never to the authors, - that I carefully search for friendly or (if I am really angry) neutral formulations for my criticism. This is how I want my reviews to be, but it can be tough to make them so. By signing, I make sure I invest the required time and energy.

Journal, conference, workshop: Differences

Computer Science is atypical among sciences in that much of our serious publishing is done at conferences and workshops, rather than only in journals as is true for most other disciplines.

Just as in any discipline, however, there are several layers of venues that require different levels of quality and originality.

• Tier 1: Top journals and top conferences expect the scientific contribution to be major (rather than small), the research to be fairly complete (journals more than conferences), the methodology to be strong and clean, and the writeup to be fully intelligible.
• Tier 2: Below that, at good journals and high-quality conferences, all this is similar, but the acceptable level of quality is somewhat lower, in particular with respect to the size of the scientific contribution.
• Tier 3: Medium-quality conferences (there are few third-tier journals in Computer Science) gradually reduce all these requirements still more, but still keep them all up in principle
• Tier 4: Workshops, in contrast, are (usually; but there are many exceptions of workshops that are in fact more like conferences) a place for incomplete results. The paper needs to describe some interesting idea, approach, or partial result, and must be intelligible, but complete scientific results are usually not required.

Make sure you read the mission statement of the journal or the call for papers of the conference or workshop to apply the right criteria.

Do not apply the same expectations to a short paper than to a full one, in particular with respect to the documentation of methodology.

Ethical aspects

• The contents of the article under review are not yet published; they should be kept confidential. If you talk to others about the material you review, keep it within your own work group and ensure the others keep confidentiality, too.
• If you fear you may have totally misunderstood a key point in an article, it is acceptable (but not required) if you contact an author for clarification.
• But note that the reviewer is typically anonymous and not all journals like it if you break this anonymity. (You may then ask the associate editor to forward your question for you)
• You must not abuse your position of relative power over the authors.
  • Put as much effort into finding strong points as you do in finding weaknesses. Control your envy (if any) of good research done by others.
  • Do not hurt the feelings of authors in cases where much criticism is due. Criticize only the work, not the people. Use neutral, unemotional terms. Be empathic; imagine yourself in their position.
• The confidentiality of article content also means you must not use ideas from the article for your own research until that article has been published. (Exception: If breaking anonymity is OK, you may ask the authors for permission. This could become the start of a cooperation sometimes.)

Comments