Not All Evidence Is Created Equal

In education, as in any field, we need to make decisions that are trustworthy. We have all seen the “educational” product that claims to be backed by evidence. But not all evidence is created equal.

Today we are going to discuss one of the main reasons a study’s results should have you raising one eyebrow: confounding variables. 

Let’s explore why it’s important to think critically about confounding variables in research, especially when it affects our students.

What Are Confounders?

Confounders are factors – other than the program or policy we are testing – that can change the results of a study. For example, let’s say two schools were part of a study on a new blended learning program. One school implemented the program, but another school didn’t. In this case, how would we know if different scores in the schools were due to the program, or the school? 

We don’t! One school might also have a 10 high-fives daily minimum that has boosted everyone’s mood and improved their test scores. We have no way of knowing. 

In this case, to remove the confounding variable we would need at least two schools in either the blended learning or alternative program.

Types of Confounds

1. Demographic Confounders: Things like age, gender, and socio-economic status can influence outcomes. A program that works for one group may not work for another if these factors aren’t considered.
2. Contextual Confounders: The school environment matters. A program might succeed in a well-resourced school but fail in one with fewer resources.
3. Temporal Confounders: When tests are given can affect results. For example, if students are tested at different times of the year, outside factors like weather can influence performance.
4. Behavioral Confounders: Students’ motivation can change outcomes. If more motivated students are the ones in a specific program, it may not reflect the program’s true effectiveness.
5. Measurement Confounders: Different tests or grading methods can lead to different results. If studies use various assessments, it can be hard to know what really improved.

Safeguarding Results from Confounders

The best way to make sure the difference between our treatment and control groups is because of the intervention is to have conditions as much like a lab as possible. 

That is, randomize assignment of the treatment, to multiple groups, at the same time. 

But for logistical and ethical reasons, these kinds of experiments are not always possible. When student learning is on the line, withholding a potentially beneficial intervention at random can be a hard sell. 

For studies that are not randomized, there are two safeguards against confounding.

Establish Baseline Equivalency

To know if one runner is faster than another runner, we need to know where the runners started. If one runner started far ahead of another, they might finish first despite being slower. In education research, making sure that two groups start at the same place is called checking the students’ or teachers’ “baseline equivalency.” To check baseline equivalency, we follow What Works Clearinghouse (WWC) standards, which state that treatment and control groups must:

“Demonstrate baseline differences are 0.05 standard deviation or smaller [or] demonstrate baseline differences are between 0.05 to 0.25 standard deviation and apply an acceptable adjustment for baseline differences” (p. 53, WWC, 2022).

If you are interested in the standardized effect size calculations or the statistical adjustments, see “General effect size formulas” of the WWC Standards and Procedures Handbook (p. 162, version 5.0). 

Final Thoughts

Anyone making decisions about students’ education should be using evidence to inform their choices. But reviewing evidence takes time, a lot of time. Meta-analyses generally at least two years. Tools like My Education Researcher cut that time down, are aligned with WWC standards, and tell you what you need to know, to make an informed decision.