Systematic Literature Reviews: An Overview

Many master’s and PhD students discover that passing their courses means writing a literature review. Sometimes that review is a standalone document; in other cases it simply sets the scene for a primary research project.

If you want to see each type in practice, start with Mishra and Nair (2015) — a clear example of a standalone review, and a textbook case of what scholars call a systematic literature review (or SLR). By contrast, Carnwell and Daly’s (2003) qualitative exploratory study briefly summarises and critiques existing findings before presenting its own primary research; this is a critical literature review (or CLR).

Clearly, then, there are different types of literature review. The less obvious — but equally important — point is that each type comes with its own advantages, disadvantages, purposes, and degrees of suitability to your own situation as a student or academic.

With that in mind, this article explains the core features of one of the most common and most powerful types of literature review: the systematic literature review. Specifically, it:

1. Outlines the main characteristics of SLRs
2. Explains their strengths and weaknesses

A full list of references (in Harvard style) is given at the end.

Systematic Literature Reviews: An Overview

According to Aveyard (2014), one of the foremost authorities on literature-review writing, systematic literature reviews are the most valid, reliable, and robust type of review a researcher can undertake.

If you are studying public health, radiology, or any other healthcare-related field, you have probably encountered SLRs already. With the growing emphasis on evidence-based practice (EBP) across the world’s healthcare systems (Marshall, 2014), the SLR serves as the principal reference point for evidence-based decision-making (Clarke and Chalmers, 2018).

So why exactly are SLRs valued so highly by healthcare professionals and policymakers? Put another way: why do SLRs consistently sit at the top of the so-called “hierarchy of evidence”? To answer that, let’s look at the SLR’s main characteristics.

Systematic Literature Reviews: Characteristics

1. Carefully selected, precisely formulated review questions (Khan et al., 2003)

The review question that guides an SLR is not plucked out of thin air. Instead, systematic processes determine what the review question will be.

For quantitative SLRs, Richardson et al.’s (1995) PICO framework is used to devise a review question; for qualitative and mixed-methods SLRs, the SPIDER framework is common (Cooke, Smith, and Booth, 2012); and for qualitative SLRs, Cleyle and Booth’s (2006) SPICE framework is popular. Another widely used option is Aslam and Emmanuel’s (2010) FINER formula.

2. Systematic literature-search strategies (Aveyard, 2014)

This characteristic gives the SLR its name, and it is one of the main reasons SLRs sit atop the hierarchy of evidence. A systematic literature-search strategy is marked by the following features:

1. Use of relevant search terms, formed with Boolean operators and keywords (Grewal, Kataria, and Dhawan, 2016);
2. Clear specification of the types of literature targeted (Wallace and Wray, 2006);
3. An unambiguous statement of which databases will be searched (Aveyard, 2014);
4. Discussion and justification of the procedures used for screening and supplementary searching, such as a manual search of reference lists (Greenhalgh and Peacock, 2005).

3. Inclusion and exclusion criteria (Patino and Ferreira, 2018)

SLRs produce valid and reliable results because they exclude studies that could undermine the veracity of their findings. Crucially, studies are not excluded arbitrarily; inclusion and exclusion criteria are established in advance.

These criteria draw on aspects of the research question (such as the population examined), the types of literature targeted, and practical considerations such as publication date ranges and language (Meline, 2006).

4. Critical appraisal (Hong and Pluye, 2018)

Every study included in an SLR must undergo a rigorous, formal, and systematic process of critical appraisal. This excludes studies of low methodological quality, maximising the validity and relevance of the SLR’s findings.

Importantly, critical appraisal must be guided by a clear framework, which safeguards against bias (Harrison et al., 2017). Prominent critical-appraisal frameworks include:

1. Critical Appraisal Skills Programme (CASP) checklists;
2. The Effective Public Health Practice Project’s (EPHPP) Quality Assessment Tool for Quantitative Studies;
3. La Torre, Backhaus, and Mannocci’s (2015) International Narrative Systematic Assessment (INSA) tool.

5. Systematic, reliable, and reproducible data extraction and evidence synthesis (Cooper et al., 2018)

Whatever studies an SLR turns up, researchers must engage with the dataset in a trustworthy, reliable, formal, and reproducible way. There are several ways to do this. For example, to extract data from a qualitative study, you might use thematic analysis as described by Nowell et al. (2017).

After data extraction, a systematic evidence-synthesis technique ensures that no relevant aspect of the dataset is overlooked. Prominent synthesis techniques include segregated, integrated, and contingent methodologies (Sandelowski et al., 2006).

6. Transparency (Aveyard, 2014)

Because SLRs form the basis of evidence-based practice, nothing should be hidden from other researchers. When an SLR is published, everything from the databases searched to the completed critical-appraisal checklists should be included.

Transparency of the search process matters too. To achieve it, authors typically publish PRISMA flow diagrams alongside their findings (PRISMA, 2009).

Systematic Literature Reviews: Strengths and Weaknesses

(a) Strengths

1. SLRs offer the strongest evidence for decision-making in many settings (Murad et al., 2016)

SLRs consistently rank at the top of the hierarchy of evidence — a direct consequence of the characteristics above, including reliability, validity, transparency, reproducibility, and robustness.

This is also why high-quality SLRs are in demand across so many fields. The practical implication for you, as an aspiring researcher, is that a well-executed SLR can improve your chances of publication in a prominent journal.

2. High-quality SLRs can significantly reduce bias (Aveyard, 2014)

Because SLRs are formal, structured, rigorous, and comprehensive, they close off many of the routes through which bias creeps in. For instance, since the researcher is not subjectively deciding which articles to include, selection bias is almost entirely eliminated (Andrews, 2017).

3. SLRs summarise large bodies of evidence (Aveyard, 2014)

A wealth of literature exists on any given topic. For busy doctors, practitioners, and policymakers, a high-quality SLR distils the findings and conclusions of a thousand articles into a single publication.

(b) Weaknesses

1. Most SLRs only consider published studies

Because unpublished studies are difficult to access, they tend to be left out of SLRs. This undermines the comprehensiveness of the review and can mean the most up-to-date evidence is missing (Müller et al., 2013).

2. SLRs are difficult to design and implement (Hagen-Zanker, McCord, and Holmes, 2011)

For novice researchers especially, the expertise required is a significant barrier. Because an SLR must produce valid and reliable evidence, the process is time-consuming and often demands collaboration between several researchers — which brings its own challenges.

3. The danger of publication bias (Müller et al., 2013)

Negative or inconclusive results often go unpublished, so the articles eventually included in an SLR may contain a disproportionate — and unrepresentative — number of positive conclusions. The widespread suppression of negative results gives rise to publication bias.

References

Andrews, L. (2017) How can we demonstrate the public value of evidence-based policy making when government ministers declare that people ‘have had enough of experts’? Palgrave Communications. 3 (11).

Aslam, S. and Emmanuel, P. (2010) Formulating a researchable question: A critical step for facilitating good clinical research. Indian Journal of Sexually Transmitted Diseases and AIDS. 31 (1), 47–50.

Aveyard, H. (2014) Doing a Literature Review in Health and Social Care: A Practical Guide. New York: Open University Press.

Carnwell, R. and Daly, W. (2003) Advanced nursing practitioners in primary care settings: an exploration of the developing roles. Journal of Clinical Nursing. 12 (5).

Clarke, M. and Chalmers, I. (2018) Reflections on the history of systematic reviews. BMJ Evidence-Based Medicine. 23, 121–122.

Cleyle, S. and Booth, A. (2006) Clear and present questions: Formulating questions for evidence based practice. Library Hi Tech. 24 (3), 355–368.

Cooke, A., Smith, D., and Booth, A. (2012) Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qualitative Health Research. 22 (10), 1435–1443.

Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., and Garside, R. (2018) Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Medical Research Methodology. 18, 85.

Greenhalgh, T. and Peacock, R. (2005) Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. British Medical Journal. 331, 1064.

Grewal, A., Kataria, H., and Dhawan, I. (2016) Literature search for research planning and identification of research problem. Indian Journal of Anaesthesia. 60 (9), 635–639.

Hagen-Zanker, J., McCord, A., and Holmes, R. (2011) Systematic Review of the Impact of Employment Guarantee Schemes and Cash Transfers on the Poor. Overseas Development Institute.

Hong, Q. and Pluye, P. (2018) A conceptual framework for critical appraisal in systematic mixed studies reviews. Journal of Mixed Methods Research.

Khan, K., Kunz, R., Kleijnen, J., and Antes, G. (2003) Five steps to conducting a systematic review. Journal of the Royal Society of Medicine. 96 (3), 118–121.

La Torre, G., Backhaus, I., and Mannocci, A. (2015) Rating for narrative reviews: Concept and development of the International Narrative Systematic Assessment (INSA) tool. Senses and Sciences. 2 (1), 31–35.

Marshall, J. G. (2014) Linking research to practice. Journal of the Medical Library Association. 102 (1), 14–21.

Meline, T. (2006) Selecting studies for systematic review: Inclusion and exclusion criteria. Contemporary Issues in Communication Science and Disorders. 33, 21–27.

Mishra, D. and Nair, S. R. (2015) Systematic literature review to evaluate and characterise the health economics and outcomes research studies in India. Perspectives in Clinical Research. 6 (1), 20–33.

Müller, K. F. et al. (2013) Defining publication bias: protocol for a systematic review of highly cited articles and proposal for a new framework. Systematic Reviews. 2, 34.

Murad, M., Asi, N., Alsawas, M., and Alahdab, F. (2016) New evidence pyramid. BMJ Evidence-Based Medicine. 21, 125–127.

Nowell, L. S. et al. (2017) Thematic analysis: Striving to meet the trustworthiness criteria. International Journal of Qualitative Methods. 16, 1–13.

Patino, C. M. and Ferreira, J. C. (2018) Inclusion and exclusion criteria in research studies: definitions and why they matter. Jornal Brasileiro de Pneumologia. 44 (2), 84.

PRISMA (2009) PRISMA Flow Diagram. Available from http://prisma-statement.org

Richardson, W. S., Wilson, M. C., Nishikawa, J., and Hayward, R. S. (1995) The well-built clinical question: A key to evidence-based decisions. ACP Journal Club. 123 (3), 12.

Sandelowski, M. et al. (2006) Defining and designing mixed research synthesis studies. Research in the Schools. 13 (1), 29.

Wallace, M. and Wray, A. (2006) Critical Reading and Writing for Postgraduates. Thousand Oaks, California: SAGE Publications.

Wildridge, V. and Bell, L. (2002) How CLIP became ECLIPSE: A mnemonic to assist in searching for health policy/management information. Health Information & Libraries Journal. 19 (2), 113–115.