Skip to Main Content

FLEX (MEDD 419, MEDD 429, MEDD 449)

Information and resources in support of medical progam FLEX projects

Introduction

Evaluating literature is a complex process requiring familiarity with the field of research, research methods and types of publications. These are skills that will continue to develop throughout your program and into your professional life.

At its core, critical evaluation means investigating quality and identifying bias. While separated out here, this is an artificial division, and many of the same critical questions address both aspects.

Take a tiered approach to your analysis:

  • Does the methodology support the research question?
  • If the methodology is sound, critique the results.
  • If the results are sound, ask, do the results support the conclusions?

Further reading: 

Subject Expertise

Growing your subject expertise will allow you to understand and critically evaluate the methods, outcomes and conclusions in a paper and evaluate its quality. While building this body of knowledge, consult with your peers, instructors and other experts.

Some of the question you will want to be asking as you read are:

Study Type
  • What type of study would best answer the authors' research question and what type of study was used?
  • Is the study longitudinal, qualitative, retrospective, and what might the implications of this be for the nature of the results?
  • Check out this post from the Centre for Evidence-Based Medicine for an overview of the pros and cons of different study types.
Methods
  • Is the sample size large enough?
  • Were reasonable steps taken to avoid bias (was a control group established, were reasonable steps taken to ensure a representative sample)?
  • Were protocols followed and did the study use established standards to test and measure?
Transparency
  • Is the raw data available for other researchers?
  • Is there enough information available to make the study reproducible?
  • Are potential conflicts of interest or funding sources listed?

Indigenous Standards Checklist

In a 2019 systematic review, Jongbloed et al. developed and used a checklist to roughly assess whether research was conducted ‘in a good way’ according to common Indigenous research standards. (Jongbloed K, Pooyak S, Sharma R, et al. Experiences of the HIV Cascade of Care Among Indigenous Peoples: A Systematic Review. AIDS and behavior. 2019;23:984-1003.)

Grey Literature

The Grey Literature for Health Sciences guide contains some guidance on evaluating grey literature. 

Research Bias

Bias is always present. Analysis should focus on how bias is identified and accounted for, not its absence.

Developing an awareness of bias, both in study design, implementation, reporting and in publishing will help with your critical evaluation of the literature.

Publishing

On an individual article level, ask if the publication is peer-reviewed, or what the quality of the journal's reputation is.

On a collective level, consider publication bias; positive results are more likely to appear in journals than negative results, impacting summative evaluations of the literature.

Methodology & Design

Do the authors discuss potential bias in the study and how they mitigated this (population selection, measurement tools...)? Is there an element of bias not identified? What might they have done differently?

Results

Are the results comprehensively reported and clearly articulated, either verbally or diagramtically? Do the authors highlight bias that may have emerged from procedural elements in the design?

Conclusions & Recommendations

The critical question here is, are the the conclusions supported by the results?

The Cochrane Handbook includes a useful overview of different types of bias: Chapter 7: Considering bias and conflicts of interest among the included studies. 

Further Reading