Introduction

Randomised Control Trials (RCTs) play a vital role in evidence-based medicine, providing valuable insights into the effectiveness of medical interventions. As medical students, it is crucial to develop the skills to critically appraise RCTs. The following article gives you a step-by-step method to systematically appraise RCTs, with real-life examples from the following study: https://www.nejm.org/doi/full/10.1056/NEJMoa2032183

A summarised RCT Appraisal Checklist is also provided at the end of this article, however we recommend reading the whole article to get a more detailed understanding on why we ask ourselves the questions provided in the checklist!

Step 1: Assess Bias

Bias: factors in the design of the RCT that can influence the end measurement and undermine the validity of the RCT

1. Selection Bias: differences in selection of who participates in the trial. (eg: selection of only people from a certain area limits the generalisability of the RCT)
2. Allocation Bias: active selection of who is allocated to the treatment or placebo groups (eg: giving healthier patients the treatment may artificially create positive data)
3. Measurement Bias: systematic errors in how the outcome is measured or reported (eg: patient on the drug may overinflate the benefit of the drug)
4. Funding Bias: if the RCT is funded by a pharmaceutical company, they may have influenced the data to make the results look more promising.

You can assess bias using the Cochrane Risk of Bias tool. This is normally used to assess bias in studies being included in a meta-analysis, however, it can give you an insight into the degree of bias in your RCT.

Step 2: Assess Study Design and Methodology

Parallel and crossover RCT designs are the most commonly used.

A parallel design involves random allocation to either placebo or treatment groups with no changes in the group. A crossover design is the same, except after a defined amount of time, the groups will enter a washout period without any intervention (to remove the effects of the previous intervention). Once complete, they will swap to the alternate intervention in the trial.

Both parallel, and crossover, designs have their own advantages and limitations as shown in the table below:

When addressing methodology, essential factors to consider are:

1. The dose, frequency, and length of the intervention – is this clinically feasible?
2. Whether there are any adjuncts to the treatments – were these in both treatment and placebo groups? Are they realistic in practise?
3. Length of trial – is it long enough to assess effects? Is the condition chronic or acute?
4. Any other factors that may influence results – the ‘ugly duckling’ effect. If anything strikes you as strange or makes you ask yourself, “why have they done this?”, it may be a confounding factor.

Step 3: Evaluate Study Population

In RCTs, the study population along with the inclusion and exclusion criteria are important to critically analyse. Understanding the characteristics of the study population, such as age, gender, disease severity, and comorbidities, helps assess how generalisable the results of the study are.

For example: if a study looks at only females over 30 years of age, then the results cannot be generalised to the entire population.

The inclusion and exclusion criteria also hint at how generalisable the study is, however, it is important to check whether all the important confounding variables have been thought of.

For example: if a study on Type 2 diabetes has ‘co-morbidities’ in the exclusion criteria, then the results of the study will not be very useful as a majority of the population with Type 2 diabetes tend to have co-morbidities such as obesity.

Finally, the study must compare the key baseline characteristics (such as age and sex) between the treatment and placebo groups to minimise any differences between the groups such that they do not interfere with the results of the study.

For example: in a study on stroke, if the mean age of participants in the treatment arm is 25 years, compared to 75 years in the placebo arm, are the results going to be valid? Look at the measured baseline characteristics and ask yourself – “Are there any other individual factors that could affect the results that the authors have not accounted for?”. If you can think of any, they may be confounding factors!

Step 4: Analyse Results

In the process of appraising an RCT, medical students should critically analyse the outcomes measured in the study. There are many questions to ask oneself when reading the results section of an RCT.

1. Do the measured outcomes actually answer the research question?

If a study is trying to assess whether a treatment reduced body weight, but the only outcome they have measured is change in LDL cholesterol, their outcome does not truly answer their original research question.

2. Did participants drop-out? If yes, why?

Participants dropping out could lead to an attrition bias which can break the initial randomisation. For example, if all the 75 years olds in the treatment group dropped out, then the average ages of the treatment and placebo groups are different. Therefore, age becomes a confounding variable!

Thinking of the causes of the attrition is also essential – did the drug have a lot of side-effects? Was the administration difficult? Was the frequency too high? Were the pills too big to swallow? There are many possibilities, but if the trial has not mentioned any then it is a red flag and may suggest the drug is unsuitable for use in the population.

3. What analysis have they used for their outcomes?

There are two main types of analysis in RCTs both of which have their advantages and limitations as shown below.

1. Intention-to-Treat (ITT): includes all participants of the trial in the analysis regardless of whether they were compliant with their treatment, recording, etc. This should be the primary analysis used in the RCT.
2. Per-Protocol (PP): only includes in the analysis the participants that followed their treatment and measurement regimen to the T.

4. Have they given any details on quality of life?

Outcome measures are well and good, but often there is no significant difference measured in the primary outcome. This does NOT mean that the treatment had no effect on quality of life!

For example, a small change in the bodyweight between treatment and placebo group may be statistically insignificant, however, it may have had an impact on the quality of life of the participants. Having this information is a massive positive in any RCT.

5. Is the difference between treatment and placebo clinically relevant?

There may be a statistically significant different in an outcome between the treatment and placebo groups in the study, but this difference may be miniscule (small differences can still be significant! It just depends on the sample size of the study).

Is this tiny difference actually useful clinically? Does it justify the cost of introducing this medication to the public?

For example, if a drug statistically significantly reduced bodyweight in the treatment group by 35 grams, will it really affect patient outcomes and clinical decision making? (Quality of life must still be assessed but it is still quite unlikely to be deemed clinically relevant).

6. Have they followed-up the results in the long term?

Many outcomes may not remain stable in the long-term. For example, bodyweight interventions are notorious for having minimal effects in the long-term or once the intervention is stopped as patients tend to gain any lost weight back. Thus, having long-term data on the participants is sometimes essential to assess whether the intervention is effective.

Conclusion

Appraising RCTs is crucial for medical students to develop evidence-based practice skills. This step-by-step guide helps evaluate studies systematically, ensuring reliable findings. We have covered the appraisal of study design and methodology, population, and results along with examples throughout to help you contextualise the information and apply it to your own RCTs. The learning points from this article are summarised in our RCT Appraisal Checklist below which will allow you to quickly keep a track of questions to ask yourself during the appraisal process.

Applying evidence-based medicine principles improves patient outcomes and enhances clinical practice so use the skills you have gained from this article to make a difference!

RCT Appraisal Checklist

Written by Nilesh Chatterjee, PhD.