Introduction to Reasearch Methodology and Program Evaluation
by Dr. David Abrahams

Validity

“We are to admit no more causes of natural things than such as are both true and sufficient to explain their appearances.”
Isaac Newton

Internal Validity and its Most Significant Threats

Threats to validity can be either internal, external, or both. A threat to validity, by definition is, any factor that influences the results of the experiment. In research and evaluation, internal validity refers to the degree the treatment or intervention effects change in the dependent variable. The greater the ability a researcher can attribute the effect to the cause, rather than to extraneous factors the higher the degree of confidence that the treatment or intervention caused the effect.

Internal validity is only relevant in studies that try to establish a causal relationship. It's not relevant in most observational or descriptive studies (Trochim, 2006). Controlling for potentially confounding variables minimizes the potential for an alternative explanation of the treatment effects. The most significant threats to internal validity are: history, maturation, testing, instrumentation, regression, selection and experimental mortality.

History

History becomes a threat when other factors external to the subjects (in addition to the treatment variable) occur by virtue of the passage of time. For example, the reported effect of a year-long, institution-specific program to improve medical resident prescribing and order-writing practices may have been confounded by a self-directed continuing-education series on medication errors provided to residents by a pharmaceutical firm's medical education liaison.

Maturation

The maturation threat can operate when biological or psychological changes occur within subjects and these changes may account in part or in total for effects discerned in the study. For example, a reported decrease in emergency room visits in a long-term study of pediatric patients with asthma may be due to outgrowing childhood asthma rather than to any treatment regimen imposed. Both history and maturation are more of a concern in children and longitudinal studies.

Testing

The testing threat may occur when changes in test scores occur not because of the intervention but rather because of repeated testing. This is of particular concern when researchers administer identical pretests and posttests. Researchers that have subjects performing skilled based task, test of memory, IQ or Manual dexterity must take the threat of testing into account when designing their research at posttest. For example, a reported improvement in medical resident prescribing behaviors and order-writing practices in the study previously described may have been due to repeated administration of the same short quiz. That is, the residents simply learned to provide the right answers rather than truly achieving improved prescribing habits.

Instrumentation

When study results are due to changes in instrument calibration or observer changes rather than to a true treatment effect, the instrumentation threat is in operation. Instrumentation is a threat when study results are due to changes in the instrument calibration or observer changes rather than to a true treatment effect; this is especially true when the measuring instruments are human observers. For example: a human observer might become more proficient as a observer, noticing patterns and nuances in an observed subject that might have existed in the pretest but are only noticing it in the posttest. As a result the observer incorrectly attributes the observed change to the treatment.

Regression

Statistical regression threat is a threat to internal validity when subjects are assigned to treatments on the basis of extreme (low and high) scores on a test. During retest, the scores of extreme scorers tend to regress toward the mean even without treatment. For example, if a group of subjects was recruited on the basis of extremely high or low scores and an educational intervention is conducted, any post intervention improvements could be due partly or entirely, to regression rather than to the educational treatments presented in the program. Conceptually, the initial extremely high test score was attributed to measurement error (represented in the variability of test scores). When this changed randomly during the next test, high scores were no longer as high as before. The result is a regression towards the mean.

Selection

The selection threat is of utmost concern when subjects cannot be randomly assigned to treatment groups, particularly if groups are unequal in relevant variables before treatment intervention. For example, one obstetrics and gynecology clinic's patients receive a pharmacy-based educational intervention and another clinic's patients receive a mailed pamphlet; both methods are designed to encourage calcium supplementation. When the outcome is measured at the end of the study, it may be confounded by the fact that the groups were not equal with respect to relevant variables (e.g., age, race, income status, hysterectomy status, and menopausal status) before the educational program was implemented.

Experimental Mortality

Experimental mortality is also known as attrition is when subjects drop out of an experiment/treatment before the study is completed. Experimental mortality is a treat to internal validity when there is a differential loss of subjects from comparison groups resulting in unequal groups (Campbell and Stanley, 1963: 5). One example is a study designed to compare the effectiveness of a drug on a randomly selected group of sick participants. One group receives the drug/treatment and the other group receives the placebo. If subjects with the most severe symptoms dropped out of the active treatment group, the treatment may appear more effective than it really is.

External Validity

External validity is the degree to which the results of the study can be generalized to a population other than those studied. External validity is widely treated as an issue to be addressed through methodological procedures. In a study, it is usually impossible to measure an entire population; as a result, measurements are taken from a sample of that population. If subjects from a sample population are not randomly selected from the population, then their particular demographic, for example there: household, age, socio-economic, ethnic, racial, religious and/or income characteristics may bias their performance and the study's results may not be applicable to the population or to another comparable group.

The purpose of research is to learn something about the behavior of people. This knowledge is useful only to the extent that we can generalize the information to a larger population. However, the more we control the environment of the subjects (sub population) in a study, the more the subjects in the experimental and control groups can become different from those in the general population. Consequently, the results may have high internal validity; they may also lack external validity, meaning that they cannot be generalized beyond the particular groups used in the experiment.

Random assignment of treatment and control groups address the threats to internal validity and often create threats to external validity. When designing an experiment, each experimenter has to decide which requirement is more important, internal or external validity, and seek a balance between the two. The designed and execution of the experiment is the most effective way of testing for the effects of one variable on another variable.

Threats to External Validity

Del Siegle, Ph.D.Neag School of Education - University of Connecticut. Web: http://www.gifted.uconn.edu/siegle/research/Samples/externalvalidity.html

Population Validity is defined as the extent to which the results of a study can be generalized from the specific sample that was studied to a larger group of subjects. They are:

The extent to which one can generalize from the study sample to a defined population. Therefore, if the sample is drawn from an accessible population, rather than the target population, generalizing the research results from the accessible population to the target population is risky.
The extent to which person logical variables interact with treatment effects. Thus, if the study is an experiment, it may be possible that different results might be found with students at different grades (a person logical variable).

Ecological Validity is defined as the extent to which the results of an experiment can be generalized from the set of environmental conditions created by the researcher to other environmental conditions (settings and conditions). They are:

Explicit description of the experimental treatment (not sufficiently described for others to replicate). If the researcher fails to adequately describe how he or she conducted a study, it is difficult to determine whether the results are applicable to other settings.
Multiple-treatment interference (catalyst effect); if a researcher were to apply several treatments, it is difficult to determine how well each of the treatments would work individually. It might be that only the combination of the treatments is effective.
Hawthorne effect (attention causes differences); subjects perform differently because they know they are being studied. "...External validity of the experiment is jeopardized because the findings might not generalize to a situation in which researchers or others who were involved in the research are not present" (Gall, Borg, & Gall, 1996, p. 475)
Novelty and disruption effect (anything different makes a difference); A treatment may work because it is novel and the subjects respond to the uniqueness, rather than the actual treatment. The opposite may also occur, the treatment may not work because it is unique, but given time for the subjects to adjust to it, it might have worked.
Experimenter effect (it only works with this experimenter); the treatment might have worked because of the person implementing it. Given a different person, the treatment might not work at all.
Pretest sensitization (pretest sets the stage); A treatment might only work if a pretest is given. Because they have taken a pretest, the subjects may be more sensitive to the treatment. Had they not taken a pretest, the treatment would not have worked.
Posttest sensitization (posttest helps treatment "fall into place"); the posttest can become a learning experience. "For example, the posttest might cause certain ideas presented during the treatment to 'fall into place' " (p. 477). If the subjects had not taken a posttest, the treatment would not have worked.
Interaction of history and treatment effect (...to everything there is a time...); not only should researchers be cautious about generalizing to other population, caution should be taken to generalize to a different time period. As time passes, the conditions under which treatments work change.
Measurement of the dependent variable (maybe only works with M/C tests); A treatment may only be evident with certain types of measurements. A teaching method may produce superior results when its effectiveness is tested with an essay test, but show no differences when the effectiveness is measured with a multiple choice test.
Interaction of time of measurement and treatment effect (it takes a while for the treatment to kick in); it may be that the treatment effect does not occur until several weeks after the end of the treatment. In this situation, a posttest at the end of the treatment would show no impact, but a posttest a month later might show an impact.

Bracht, G. H., & Glass, G. V. (1968). The external validity of experiments. American Education Research Journal, 5, 437-474.

Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction. White Plains, NY: Longman.

Best,