Non-randomized studies and natural experiments
From ICE Primer: A Tobacco Control Research Methodology Primer
The gold standard for comparative studies of two or more groups of subjects is generally accepted to be the randomized trial. Randomization of subjects to comparison conditions can be done at the level of the individual, as in the classical randomized controlled trial (RCT), or at the level of the cluster as in a cluster randomized trial. Randomization helps to control for the effects of confounding factors by ensuring that an impartial means, (viz. the randomization device), has assigned the subjects to the comparison conditions. With randomized studies, we have greater confidence that any observed intervention effect is causal. This is particularly true if the effect is seen across more than one study (i.e. the effect is replicable).
The terms "observational studies" and "quasi-experimental designs" are often used to label investigations that have not involved random assignment to conditions. In many situations, randomization is not feasible (or ethical), the cost of a randomized trial may be prohibitive, or data might already be available because of efforts that took place outside of a research study. The term "natural experiment" is frequently used to denote this latter type of "study".
The accepted link between cigarettes and lung cancer is one example of a causal relationship that has evolved from a series of non-randomized studies in humans.
For example, in comparing the effects of a provincial tobacco policy, it is not feasible either to randomly assign the strategy to a province or to randomly assign individuals to a province. For a community-based intervention, recruiting sufficient communities, randomly assigning the communities to conditions, ensuring that the communities follow the study protocols, and following individuals in the communities over enough time to ensure the intervention has had time to have had an effect, make such studies expensive and difficult to conduct. Analysis of specific areas, schools, etc. may reveal jurisdictions where smoking rates are unusually low. Identifying these jurisdictions and studying the interventions in detail could reveal promising initiatives in tobacco control.
In designing a comparative study that cannot use randomization, there are some obvious issues. The same issues will apply when assessing a non-randomized study. These issues include:
- Is there a comparison (control) group? There needs to be a collection of subjects to which we can compare our intervention subjects. In this way, we are not attributing changes to the intervention when there are similar changes in subjects who did not receive the intervention.
- Is the control group similar with respect to factors that might be related to the response? If there are important factors that are confounded with the presence of the intervention, then differences could be due to these factors and not the intervention.
- Is there replication? Demonstrating that the intervention has an effect in more than one place, when delivered by different providers, to different subjects, etc. imparts greater confidence that the effect is real.
