The Reviews I Wish to Get

As a researcher, I submit research articles for peer-review. The reviews I get are unfortunately too often deceptive—not only because most are rebuttals. These reviews are of very little use, beyond improving my broken English.

Below are the key points I wish reviewers would discuss. As a matter of fact, I do also consider them when I write reviews. While I am aware of the time it takes to write a detailed review, I am convinced that it does help moving research in the right direction.

First, the “Science”

Are my Objectives Clear?

First, I would like to know if the problem I tackle, and if my motivations make sense.

1. Do you understand the problem? I often spend times crafting an introduction that explains my problem, and why existing approaches are not sufficient.

2. Do you understand why I am doing this? I also strive to explain why solving this problem matters, and how solution may in turn, contribute to larger or even societal challenges?

3. Do you think it is worth? Should you disagree with the benefits I foresee, please explain me what you disagree with.

4. Have you grasped the general idea? Eventually, I hope you understand how the big pieces fit together.

Is it Really New?

Knowing the state-of-the-Art is hard, so next comes the question of novelty. I do my best to survey and understand existing literature, and yet, I will always overlook some references, especially in other fields.

1. Do you know anything similar? My idea may not be as new as I think, and other may have already experiment with it. I found it very difficult to spot articles from other fields that uses a completely different parlance. If you do, please give me one exact reference.

2. Do you know anything equivalent? It may be as well that others already had different and maybe better ideas, but I haven’t stumbled upon. If you know some, please let me know.

3. Is this a sufficient increment? From time to time, I extend some previous publications. While I roughly target 50 % of new material, this remains a matter of interpretation.

The Magic

OK, now we have got the problem, the motivation and the overall approach, let’s dive into the details.

1. Is it correct? In other words, does it address the problem I described. I try my best, but there may always be cases I have not thought of.

2. Would you have done it the same way? Maybe you are thinking of a better idea, or you would have tried something else?

Would you Buy it?

My next questions concerns the evaluation, that is how what I propose solves the problem. Depending on the state-of-the-Art and how mature is my research, I may make three different claims:

1. It is possible! It may sound naive, but not every idea is practical—at least not all my ideas. Building a prototype that implements my solution is the first step.

2. It works! My prototype is operational, but, it must also solve problems, from toy examples to real-life case-studies, ideally.

3. It is better! My solution works but it must also costs reasonable resources (time, memory, etc.), and ideally outperform its competitors.

So, will you buy any of these? I guess it depends on the experiments I describe to support them. For each, I wish to know:

1. Is it the right experiment? Do you agree that my experiment tests one of these claims, or did I get confused and am I testing something slightly different. Maybe you would have come up with a different experiment?

2. Do you see any bias? Am I making any additional assumptions in my experiment that reduce its applicability?

3. Could you reproduce it? Ideally, I should provide enough material for you to reproduce my experiment. Do you feel confident you have enough details and material available?

These experiments are just a means to produce the data that eventually confirm or disprove my claims.

1. Would you draw the same conclusions? It is difficult to be completely objective in how we interpret the data. Do you reach the same conclusions as I?

2. Is it statistically convincing? Depending on the type of experiment, I may use statistics quantify evidence—not necessarily to mislead you I promise. Do think it helps? Would you have presented other numbers?

3. Should more data be available? If my experiment seems difficult to reproduce, would you like to see additional data available.

Only then, the form

If you still have time and energy, I appreciate it. Now, I would like be happy to know how you feel about the text.

The Flow

1. Is it clear? Did you spend 5 hours reading again and again the same single paragraph to eventually gave up? What did you miss to understand? Are there some magical steps? Is there too little or too much information?

2. Is the flow coherent? Did you have to go back and forth while reading the paper or did I manage to the right information in the right place?

3. Is it self-contained? Did you miss something or did you have to read additional resources to understand?

4. Diagrams, Tables and charts? Did you find them useful or useless? Maybe some text would better be a chart or diagram.

Style and Typos?

You are still there, so I must thank you for your effort. So here are my last two questions, the style and the grammar.

1. The style? Did you find that the text reads well? Are sentences clumsy, too long. Maybe you have found the text too academic or, at the opposite, too colloquial.

2. Any Grammar or Spelling mistake? I know myself, and there must be spelling mistakes and typos lurking here and there. Thank you for pointing the ones you spotted.

All your comments will help me rework my articles and hopefully we shall together save some time for my next reviewers.