Internal Validity in Impact Evaluation: Overview, Importance, and Ways for Improvement

The soul of impact evaluation is to ascertain the treatment effect of a specific intervention. In essence, one must establish whether or not one received outcomes and whether or not those outcomes were the product of actions, conditions, or other intervention variables. To arrive at a credible response, it is necessary to determine if the observed outcomes were caused by the intervention or by other causes. It’s easier said than done, and the most important roadblock in attribution is the construct of threat to internal and external validity.  

What is internal validity? 

Internal validity is the ability to establish a causal relationship between treatment and outcome variables. It also reflects a researcher’s confidence that an experimental treatment or procedure caused the observed change in the dependent variables. This very confidence is called internal validity and is threatened by any factor that undermines this confidence.  

But what are the threats to internal validity?

Such threats happen during the study and make you wonder if your intervention made a difference or something else did. Often, the threats come from parts running simultaneously and could affect the problem you’re trying to solve. Without a way to tell the difference between their effects and the effects of your program, you won’t be able to tell if the changes you see were caused by your work or by one or more of these other factors. 

Regression towards the mean: This term describes how a measure’s most extreme and lowest-performing results tend to be closer to the average group over time. You can start a program with people with low or very high levels of the measure you are considering, such as maternal health outcome, knowledge, attitude, and certain behaviors toward people of other races or backgrounds. Their scores could be closer to the average, even if they are not using any program. 

Selection bias: People who select participants might favor a group that is more likely to change than the cross-section of the population. It is why employment training programs that are paid according to how many people they help find jobs are so popular. They will choose those with the most skills to do the job, and they are more likely to get hired than those with fewer skills.  

Attrition: Losing people from your research group is undesirable because it creates a bias. When people drop out of your research, your results will depend on those who stay. It will hurt the credibility of your research because you won’t be able to draw as many conclusions from it. 

Historical events: An event from the past may directly or indirectly affect the research results through the people who participate. Some things, like natural disasters or politics, can make it hard for the people participating in the research to work together and affect how well they do. 

Instrumentation: Depending on your research method, you can tell people participating in your study how to act. It could make your participants do things differently than they would have. 

Maturation: This means that researchers should consider timing a very important variable. If your research participants grew older or underwent a biological (time) change, it may be hard to show that time did not affect the study results. 

History: History is a threat to the internal validity of an experiment. History is any event besides the independent variable that happened during or outside the experiment and that which could explain the results. It talks about the effects of things that happen to people every day. Changes in the weather, the news, or the participants’ lives could affect how well they did in an experiment. In our example of losing weight, the subject’s eating habits and activity level outside the experiment could affect how much weight they lose or gain. 

Testing: This happens to our test scores when we retake a test or study for it. The second test score could be affected by how well the person knows the test. The final score may change if the test is taken more than once. 

Confounding: This is when changes in an outcome variable could have been caused by a third variable that has something to do with the treatment you gave. 

What makes internal validity stronger? 

If you want to improve the internal validity of a study, you should think about the parts of your research design that make it easier to rule out other possible explanations. There are many ways to improve the internal validity of a study. 

Blinding: When participants and sometimes researchers don’t know what intervention they’re getting (like when a placebo is used in a drug study) so that this information doesn’t change how they think and act, it could change the study results. 

Experimental manipulation: This is changing an independent variable in a study, like giving smokers a program to help them quit, instead of just observing a relationship without making any changes (examining the relationship between exercise and smoking behavior). 

Choosing at random: Choosing your participants randomly or in a way that shows how they fit into the group you want to study. 

Randomization: Putting people in treatment and control groups by chance and ensuring there is no systematic bias between the groups can help with strengthening internal validity.  

Kultar Singh – Chief Executive Officer, Sambodhi