Ten Simple Rules for Writing a Literature Review

In today's scientific community, the demand for literature reviews is rapidly growing. This demand stems from the explosive growth in scientific publications [1]. For instance, the number of papers indexed in the Web of Science database in 2008 on malaria, obesity, and biodiversity increased by three, eight, and forty times respectively compared to 1991 [2]. Faced with such a massive volume of papers, scientists cannot possibly meticulously review every new piece of literature relevant to their research [3]. Therefore, regularly summarizing the latest literature is both advantageous and increasingly necessary. Although scientists primarily gain recognition through original research, timely literature reviews can not only bring new synthetic insights but often attract a wide readership [4]. However, for these summaries to truly serve their purpose, they need to be written professionally [5].

Starting a literature review from scratch can be a daunting task. This is why researchers who have long focused on a particular research question are best suited to write literature reviews in their field. Today, some graduate schools offer courses on literature reviews, as most graduate students often start their projects by reviewing existing research [6]. However, it's likely that most scientists haven't systematically thought about how to conduct a literature review.

Writing a literature review requires the integration of multiple skills, from finding and evaluating relevant materials to synthesizing information from different sources, from critical thinking to interpretation, evaluation, and citation techniques [7]. In this article, I will share ten simple principles derived from my experiences participating in about 25 literature reviews as a doctoral student and postdoc. These ideas and insights also come from discussions with co-authors and colleagues, as well as feedback from reviewers and editors.

Rule 1: Define the Topic and Target Audience

How do you choose a review topic? There are too many questions worth exploring in contemporary science, and you might spend a lot of time attending conferences and reading literature to think about what to review. On one hand, if you spend years choosing, you might find that others have beaten you to it. On the other hand, only well-thought-out topics are likely to produce excellent literature reviews [8]. A good topic should at least:

Interest you (ideally, you should encounter a series of recent papers related to your work that need critical summarization),
Cover an important aspect of the field (so that many readers will be interested in the review, and there's enough material to write about),
Focus on a clearly defined question (otherwise you might cover thousands of publications, making the review lose focus).

Potential review ideas might come from papers that provide lists of key research questions [9], or they might arise from flashes of insight during daily reading and discussions. In addition to choosing a topic, you should also identify the target audience. In many cases, the topic (such as network services in computational biology) automatically defines the audience (such as computational biologists), but the same topic might also interest neighboring fields (such as computer science, biology, etc.).

Rule 2: Search and Re-search the Literature

After determining the topic and target audience, start searching for literature and downloading relevant papers. Here are five suggestions:

Keep a record of the search terms you use (so that your search process can be reproduced [10]),
Keep a list of papers whose full text you cannot temporarily access (so you can retrieve them later through other means),
Use reference management software (such as Mendeley, Papers, Qiqqa, Sente),
Develop some criteria for excluding irrelevant papers early on (these criteria can later be described in the review to define the scope of the research),
Do not just search for research papers in the field you plan to review, but also seek out published related reviews.

It's likely that someone has already published a literature review (Figure 1), if not exactly on the question you plan to explore, at least on a related topic. If there are already several or many reviews of your question, my advice is not to give up, but to carry on with your own literature review,

Discuss the approach, limitations, and conclusions of previous reviews in your review,
Try to find a new angle that previous reviews haven't adequately covered,
Incorporate new material that has accumulated since the publication of previous reviews.

When searching for relevant papers and reviews, the usual rules apply:

Be thorough,
Use different keywords and databases (such as DBLP, Google Scholar, ISI Proceedings, JSTOR Search, Medline, Scopus, Web of Science),
Look at who has cited past relevant papers and book chapters.

Rule 3: Take Notes While Reading

If you read all the literature first and then start writing the review, you'll need exceptional memory to recall the content of each paper, as well as the impressions and associations you had while reading. My advice is to start recording interesting information snippets, insights on how to organize the review, and writing ideas while reading. This way, when you finish reading your selected literature, you'll already have a draft of the review.

Of course, this draft will still need extensive revision, restructuring, and deep thinking to form a text with coherent arguments [11], but you'll avoid facing a blank document. Be careful with citations when taking notes, and if you temporarily copy directly from the literature, be sure to rephrase in your own words in the final draft. Pay attention to referencing from the start to avoid incorrect citations. Using reference management software will save you a lot of time.

Rule 4: Choose the Type of Review

After reading the literature and taking notes, you'll have a rough idea of the amount of material available for the review. This might be a good time to decide whether to write a mini-review or a comprehensive review. Some journals now prefer to publish shorter reviews focusing on research from recent years, limiting word count and number of citations. Mini-reviews are not secondary reviews: they might attract more attention from busy readers, although they inevitably simplify some issues and omit some relevant material due to space constraints. The advantage of comprehensive reviews is that they have more freedom to explore the complexities of specific scientific developments in detail, but they might end up in the pile of "important papers to read" for readers who don't have much time to read monographs.

There might be a continuum between mini-reviews and comprehensive reviews. The dichotomy between descriptive reviews and integrative reviews also applies. Descriptive reviews focus on the methodology, findings, and interpretation of each reviewed study, while integrative reviews attempt to find common ideas and concepts from the reviewed material [12]. There's a similar distinction between narrative reviews and systematic reviews: narrative reviews are qualitative, while systematic reviews attempt to test hypotheses based on published evidence, which is collected using predefined protocols to reduce bias [13],[14]. When systematic reviews analyze quantitative results in a quantitative way, they become meta-analyses. The choice between different types of reviews depends on the specific situation, not only on the nature of the collected material and the preferences of the target journal but also on the available time and the number of coauthors [15].

Rule 5: Keep the Review Focused, but Make it of Broad Interest

Whether you plan to write a mini-review or a comprehensive review, it's wise to keep it focused [16],[17]. It's easy to try to include material just for the sake of inclusion, leading to a review that attempts to cover too much at once. The need to keep the review focused can be challenging for interdisciplinary reviews, as their purpose is to bridge gaps between different fields [18]. If you're writing a review on how epidemiological methods are used to model the spread of ideas, you might be inclined to include material from both parent fields, namely epidemiology and cultural transmission studies. This might be necessary to some extent, but in this case, a focused review would only deal in detail with those studies at the intersection of epidemiology and idea transmission.

While focus is an important feature of a successful review, this requirement must be balanced with the need to make the review appealing to a wide audience. This paradox can be resolved by discussing the broader implications of the reviewed topic for other disciplines.

Rule 6: Be Critical and Consistent

Reviewing literature is not a simple collection task. A good review not only summarizes the literature but also critically discusses it, identifies methodological problems, and points out research gaps [19]. After reading a literature review, readers should have a rough understanding of:

The main achievements in the field
The main areas of controversy
The outstanding research questions

Reviews that succeed in all these aspects are challenging. One solution is to invite a group of complementary coauthors: some are good at outlining achievements, some are good at identifying future challenges, and some have insight into predicting solutions. If your research group happens to have such a team, you should definitely start writing a literature review! In addition to critical thinking, literature reviews also need to maintain consistency, for example, in choosing between passive voice and active voice, present tense and past tense.

Rule 7: Find a Logical Structure

Like a well-baked cake, a good review has some notable features: worth the reader's time investment, timely, systematic, well written, focused, and critical. It also needs a good structure. For reviews, the usual structure of introduction, methods, results, and discussion for research papers is rarely used or applicable. However, a general introduction to the background and a restatement of the main points and key information covered at the end are still meaningful for reviews. For systematic reviews, there's a trend to include information on how the literature was searched (databases, keywords, time limits) [20].

How do you organize the body of the review to attract readers and guide them through the entire text? Usually, it's helpful to draw a concept map of the review, for example, using mind mapping techniques. This diagram can help identify logical ways to organize and connect different parts of the review [21]. This is not only useful during the writing phase but also helpful for readers if the diagram is included as an illustration in the review. Carefully selected charts and graphics relevant to the reviewed topic can also help structure the text well [22].

Rule 8: Make Use of Feedback

Literature reviews are usually peer-reviewed in the same way as research papers, which is correct [23]. As a general rule, incorporating reviewers' feedback greatly helps improve review drafts. After reading the review with fresh eyes, reviewers might detect inaccuracies, inconsistencies, and ambiguities that the authors have overlooked due to reading it multiple times. However, it's advisable to carefully read the draft again before submission, as last-minute corrections of typos, logical jumps, and confused sentences can allow reviewers to focus on providing suggestions on content rather than form.

Feedback is crucial for writing excellent reviews and should be sought from various colleagues to get diverse perspectives on the draft. This might sometimes lead to conflicting views on the merits of the paper and how to improve it, but such a situation is better than no feedback. Diverse feedback on literature reviews can help identify where the consensus view sits in the landscape of the current scientific understanding [24].

Rule 9: Include Your Own Relevant Research Objectively

In many cases, the reviewer of the literature will have published research relevant to the review they are writing. This can create a conflict of interest: how can reviewers report objectively on their own work [25]? Some scientists might be overly enthusiastic about what they have already published, so they might tend to overemphasize their own findings in the review. However, bias can also occur in the opposite direction: some scientists might tend to downplay their own achievements, thus tending to understate their contribution to a field when reviewing it (if any).

In general, a literature review should neither be a self-promotional brochure nor an exercise in competitive self-denial. If reviewers can competently produce a well-organized, clear, fluent, and valuable review for readers, they should be able to objectively review their own relevant findings. In reviews written by multiple authors, this can be achieved by assigning the review of the results of one author to other co-authors.

Rule 10: Keep Up with Recent Developments, but Don't Ignore Earlier Studies

Given the accelerating pace of scientific paper publication, today's literature reviews need not only to be aware of the overall direction and achievements of a research field but also to keep up with the latest research trends to avoid being outdated upon publication. Ideally, a literature review should not identify major research gaps that have just been addressed in a series of papers about to be published (of course, this applies equally to earlier overlooked studies, the so-called "sleeping beauties" [26]). This means that literature review authors should preferably pay attention to electronic preprints of papers about to be published, as these papers might take several months to appear in scientific databases. Some reviews will state that they have scanned the literature up to a certain point in time, but given that peer review can be quite a lengthy process, it might be worth conducting a comprehensive search for newly emerged literature during the revision stage.

It's particularly challenging to evaluate the contribution of papers that have just emerged, as there's little perspective to measure their importance and impact on further research and society. Inevitably, new papers on the reviewed topic (including independently written literature reviews) will appear from various directions soon after the review is published, so updated reviews might be needed very quickly. But this is the nature of science [27]–[32]. I wish everyone good luck in writing literature reviews.

Acknowledgments

Many thanks for insights and discussions to M. Barbosa, K. Dehnen-Schmutz, T. Döring, D. Fontaneto, M. Garbelotto, O. Holdenrieder, M. Jeger, D. Lonsdale, A. MacLeod, P. Mills, M. Moslonka-Lefebvre, G. Stancanelli, P. Weisberg, and X. Xu, and to P. Bourne, T. Matoni, and D. Smith for helpful comments on a previous draft.

Funding Statement

This work was funded by the French Foundation for Research on Biodiversity (FRB) through its Centre for Synthesis and Analysis of Biodiversity data (CESAB), as part of the NETSEED research project. The funders had no role in the preparation of the manuscript.

References

Rapple C (2011) The role of the critical review article in alleviating information overload. Annual Reviews White Paper. Available: http://www.annualreviews.org/userimages/ContentEditor/1300384004941/Annual_Reviews_WhitePaper_Web_2011.pdf. Accessed May 2013.
Pautasso M (2010) Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics 85: 193–202 doi:10.1007/s11192-010-0233-5
Erren TC, Cullen P, Erren M (2009) How to surf today's information tsunami: on the craft of effective reading. Med Hypotheses 73: 278–279 doi:10.1016/j.mehy.2009.05.002
Hampton SE, Parker JN (2011) Collaboration and productivity in scientific synthesis. Bioscience 61: 900–910 doi:10.1525/bio.2011.61.11.9
Ketcham CM, Crawford JM (2007) The impact of review articles. Lab Invest 87: 1174–1185 doi:10.1038/labinvest.3700688
Boote DN, Beile P (2005) Scholars before researchers: on the centrality of the dissertation literature review in research preparation. Educ Res 34: 3–15 doi:10.3102/0013189X034006003
Budgen D, Brereton P (2006) Performing systematic literature reviews in software engineering. Proc 28th Int Conf Software Engineering, ACM New York, NY, USA, pp. 1051–1052. doi:10.1145/1134285.1134500.
Maier HR (2013) What constitutes a good literature review and why does its quality matter? Environ Model Softw 43: 3–4 doi:10.1016/j.envsoft.2013.02.004
Sutherland WJ, Fleishman E, Mascia MB, Pretty J, Rudd MA (2011) Methods for collaboratively identifying research priorities and emerging issues in science and policy. Methods Ecol Evol 2: 238–247 doi:10.1111/j.2041-210X.2010.00083.x
Maggio LA, Tannery NH, Kanter SL (2011) Reproducibility of literature search reporting in medical education reviews. Acad Med 86: 1049–1054 doi:10.1097/ACM.0b013e31822221e7
Torraco RJ (2005) Writing integrative literature reviews: guidelines and examples. Human Res Develop Rev 4: 356–367 doi:10.1177/1534484305278283
Khoo CSG, Na JC, Jaidka K (2011) Analysis of the macro-level discourse structure of literature reviews. Online Info Rev 35: 255–271 doi:10.1108/14684521111128032
Rosenfeld RM (1996) How to systematically review the medical literature. Otolaryngol Head Neck Surg 115: 53–63 doi:10.1016/S0194-5998(96)70137-7
Cook DA, West CP (2012) Conducting systematic reviews in medical education: a stepwise approach. Med Educ 46: 943–952 doi:10.1111/j.1365-2923.2012.04328.x
Dijkers M (2009) The Task Force on Systematic Reviews and Guidelines (2009) The value of "traditional" reviews in the era of systematic reviewing. Am J Phys Med Rehabil 88: 423–430 doi:10.1097/PHM.0b013e31819c59c6
Eco U (1977) Come si fa una tesi di laurea. Milan: Bompiani.
Hart C (1998) Doing a literature review: releasing the social science research imagination. London: SAGE.
Wagner CS, Roessner JD, Bobb K, Klein JT, Boyack KW, et al. (2011) Approaches to understanding and measuring interdisciplinary scientific research (IDR): a review of the literature. J Informetr 5: 14–26 doi:10.1016/j.joi.2010.06.004
Carnwell R, Daly W (2001) Strategies for the construction of a critical review of the literature. Nurse Educ Pract 1: 57–63 doi:10.1054/nepr.2001.0008
Roberts PD, Stewart GB, Pullin AS (2006) Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine. Biol Conserv 132: 409–423 doi:10.1016/j.biocon.2006.04.034
Ridley D (2008) The literature review: a step-by-step guide for students. London: SAGE.
Kelleher C, Wagener T (2011) Ten guidelines for effective data visualization in scientific publications. Environ Model Softw 26: 822–827 doi:10.1016/j.envsoft.2010.12.006
Oxman AD, Guyatt GH (1988) Guidelines for reading literature reviews. CMAJ 138: 697–703.
May RM (2011) Science as organized scepticism. Philos Trans A Math Phys Eng Sci 369: 4685–4689 doi:10.1098/rsta.2011.0177
Logan DW, Sandal M, Gardner PP, Manske M, Bateman A (2010) Ten simple rules for editing Wikipedia. PLoS Comput Biol 6: e1000941 doi:10.1371/journal.pcbi.1000941
van Raan AFJ (2004) Sleeping beauties in science. Scientometrics 59: 467–472 doi:10.1023/B:SCIE.0000018543.82441.f1
Rosenberg D (2003) Early modern information overload. J Hist Ideas 64: 1–9 doi:10.1353/jhi.2003.0017
Bastian H, Glasziou P, Chalmers I (2010) Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 7: e1000326 doi:10.1371/journal.pmed.1000326
Bertamini M, Munafò MR (2012) Bite-size science and its undesired side effects. Perspect Psychol Sci 7: 67–71 doi:10.1177/1745691611429353
Pautasso M (2012) Publication growth in biological sub-fields: patterns, predictability and sustainability. Sustainability 4: 3234–3247 doi:10.3390/su4123234
Michels C, Schmoch U (2013) Impact of bibliometric studies on the publication behaviour of authors. Scientometrics doi:10.1007/s11192-013-1015-7. In press.
Tsafnat G, Dunn A, Glasziou P, Coiera E (2013) The automation of systematic reviews. BMJ 346: f139 doi:10.1136/bmj.f139
Pautasso M, Döring TF, Garbelotto M, Pellis L, Jeger MJ (2012) Impacts of climate change on plant diseases - opinions and trends. Eur J Plant Pathol 133: 295–313 doi:10.1007/s10658-012-9936-1