Statement of Gale E. Grant, Director, Abstinence Education Initiative,
Virginia Department of Health, Richmond Virginia

Testimony Before the Subcommittee on Human Resources
of the House Committee on Ways and Means

Hearing on Teen Pregnancy Prevention

November 15, 2001

Introduction

In general, evaluation research and its findings serve three primary functions:

1. to judge merit or worth
2. to improve programs and policies
3. to generate knowledge.

Research should never be undertaken to ‘prove’ something - research probes. A substantive finding or hypothesis is one that repeatedly survives such probing. A single piece of work should never be looked upon as either complete or conclusive.  In order to make any kind of conclusive statements about the function, efficacy, and/or contributions of abstinence education, there must exist a body of literature.

The literature on the effectiveness of abstinence education programs is meager at best.  None of the small number of published studies have demonstrated reductions in sexual activity levels, but each study suffers design flaws that prevent conclusions about either positive or negative effects.   Thus, we presently have no scientific basis for judging the merit or worth of such programs, for improving these programs, or for developing policies related to these programs.

Is Rigorous Evaluation of Abstinence Education Programs Possible?

The strength or rigor of any program evaluation research is dependent in large part upon the following contributing factors:

1) The strength and integrity of the program that is being evaluated
2) The strength of the research design/methodology
3) The use of assessment instruments whose measures are both reliable (consistent) and valid (accurate)
4) The replicability of the research findings

Each of these contributing factors are controllable, thus making rigorous evaluation of abstinence education programs theoretically possible.

What are the Challenges to the Rigorous Evaluation of Abstinence Education Programs?

By its very nature, human subjects research occurs outside of a controlled laboratory environment.  Programs are rarely implemented exactly as they are intended, unanticipated outside influences often come into play, etc.  The accuracy and completeness of documentation of the processes and events that took place during the period of the study are crucial for the interpretation of research data.

The use of strong research designs is often hampered by lack of resources (evaluation is expensive), political pressures (rarely do people want to serve as the control/comparison group and not receive the intervention), structural limitations (class assignments and student schedules), and poor planning (in most cases, evaluation is an after-thought).  There are four primary designs for measuring program outcomes and impacts.  The first two are the only ones which allow for true assessments of program outcomes/impacts:

1) Random Assignment/Experimental Design

-  This is the strongest design available because it eliminates all sources of bias.
-  This design must be developed prior to program implementation

2) Comparison Group with Pre and Post Measures/Quasi-Experimental  Design

-  Next strongest design
-  The primary limitation to this design is that it does not control for pre-existing differences in unmeasured attitudes/values/behaviors/risk factors

3) Comparison Group with Post Measures Only

- Has serious limitations since it runs the risk of peer group selection biases and does not control for pre-existing differences in measured or unmeasured attitudes/values/ behaviors/risk factors

4) Pre-post test with Program Participants Only

- Has serious limitations because it does not control for maturation

The definition of abstinence and abstinence education is often confusing/ambiguous.  In addition, consensus regarding program goals and outcomes is not always easy to come by.  For example, decreases in sexual activity or delays in the initiation of sexual activity are definitely seen as positive outcomes.  However, the definition of sexual activity (intercourse versus other forms of sexual involvement) is frequently a subject of debate.

Different types of knowledge are generated based on the type of evaluation research being conducted.  For example, formative evaluation research assists programs with the documentation of program processes and their implementation.  This leads to programs that are more effective.  Summative evaluation research, on the other hand, assists sponsors with information about program success/effectiveness.  This leads to greater accountability for resources and more effective policy decision-making.

It has been said by opponents of abstinence education that the efficacy of it has not been demonstrated.  In fact, opponents have attempted to say that it does not work.  The truth is that the literature on the efficacy of abstinence education programs is meager at best, and that the jury is still out on whether or not it is effective. Where there is literature on the efficacy of abstinence education programs, that literature has historically been replete with methodological weaknesses.  Many of these methodological problems were due to compromises of program integrity from weak or poor program design/implementation resulting from inadequate funding.

History

The Virginia Abstinence Education Initiative is a five-year, multi-component effort to implement new approaches that will help adolescents develop the attitudes and skills necessary to delay sexual involvement until marriage, and to evaluate systematically the effectiveness of those approaches. Unlike many of the evaluation of abstinence education efforts around the country both past and present, systematic evaluation of the program was built into the Virginia Abstinence Education Initiative (VAEI) from the very beginning.  Due to the criticisms thrown at abstinence education programs, the VAEI sought two things as a priority: 1) adequate funding to support strong program design and integrity of program implementation and 2) adequate funding to support formative and summative program evaluation.

Consequently, the Virginia Department of Health (VDH), which has the responsibility for VAEI program administration, built evaluation expectations into its Request for Proposals.  In addition, VDH established an Evaluation Consortium comprised of faculty from five public universities in Virginia (University of Virginia, George Mason University, James Madison University, Virginia Commonwealth University, Christopher Newport University) with expertise in program evaluation and one national expert on the evaluation of abstinence education programs.  The Evaluation Consortium provides technical assistance to local program sites, provides guidance around the design of data collection and evaluation methodology, and data analyses and interpretation.  In addition, VDH has subcontracted with the Survey and Evaluation Research Laboratory (SERL) at Virginia Commonwealth University to design and implement a data reporting system to support evaluation and monitoring activities.

Overview

The VAEI evaluation system is comprised of both formative and summative evaluation components.  Data for the VAEI evaluation system is collected using the the following five tools:

1.     Quarterly Implementation Progress Reports (QIPRs): The QIPR serves as a qualitative report on each program's activities and barriers related to achieving the overall program goals.  The QIPR is used to record the history of the program, including any events that occur in the school or community that may influence the participants in the program.

2.     Community Education Information Reports (CEIRs): The CEIR serves as a way to capture basic information on activities and audiences that are very diverse in nature.  Community education is defined as a one-time or short-term program where it is impractical or unfeasible to capture attendance data (or for short series of sessions where there is no expectation that the same participants will return for each session).

3.     Intervention Project Attendance Reports (IPARs): Intervention projects are defined as projects where there is an expectation that individuals will be "enrolled" into a planned approach or curriculum that includes multiple contacts where the information in each subsequent session builds upon information that has been covered previously. 

4.     Survey of Youth Attitudes and Behaviors: the purpose of the survey is to capture the attitudes and behaviors of youth related to marriage, sex, and sexual abstinence.  This questionnaire is administered to all participants at the first or second session (pre-) and at the final session (post-) to assess the level of impact of the program's activities.  The questionnaire is also administered to program participants annually over the course of program funding (longitudinal design).  This longitudinal design allows for the capturing of both long and short term changes.  Additionally, in order to attribute any change to the program's activities, the same survey is also administered to a comparison group within two weeks of the participant administrations.  This quasi-experimental design helps to insure that any changes noted pre-to-post program can be attributed to the intervention and not due to normal maturation or other events that may happen in the environment.  Since this initiative is implemented over five years, the longitudinal and quasi-experimental nature of the design creates a rather complex but rich source of data.  (see Table 1).

5.     Other Methods as needed as determined cooperatively between the program site, the evaluation consortium member assigned to that site, in consultation with all members of the Evaluation Consortium.

Preliminary Findings

Six program sites were selected to receive VAEI funding during the first year of the initiative.  By nature of human subjects research that is outside of a controlled laboratory environment, there will never be a perfect study.  However, results from the first year of a five-year study show strong scale reliability and strong comparability between program and control groups in all but one of the six sites.  Having accurate and complete recording of processes and events that take place during the period of the study, a strong design/methodology from the very beginning, and the use of reliable/valid assessment instruments establishes a good foundation for a strong study.

In addition, during the first year, three of the six funded sites had enough participants and strength and integrity of program implementation to warrant some in-depth analyses about short-term program effects.  Of the three sites, one had very strong statistically significant short term treatment effects in the desired direction, one had moderate statistically significant short term treatment effects in the desired direction, and one showed little significant short term treatment effects in the desired direction.   These types of outcomes are generally not attainable in the first year of a pilot project since first year projects generally have weak program elements and problematic rates of participation.  On the flip-side, three of the six sites did experience the expected problems related to weak program implementation or less than optimal participation rates.  However, it is anticipated that with the feedback and lessons learned from the first year, these already positive findings will become increasingly so over the next three years.

The strength of the VAEI design not only shows great promise in its ability to contribute to the body of knowledge about the efficacy of abstinence education programs, but has also attracted national attention and recognition.  Due to the groundwork laid by the VAEI, one of the six VAEI program sites has been selected by Mathematica Policy Research (MPR) as a model program site for their federally funded national evaluation of abstinence education programs.

Data Summary for Years 1 and 2 of 5

• Year 1 data showed very strong pre and post-test linkages in 3 of 6 programs.  Year 2 data showed very strong pre and post-test linkages in all programs being evaluated.
• Year 1 and Year 2 data show strong comparability between program and control groups in more than half of the programs being evaluated.  Some work remains to be done to increase group comparability in two of the sites.
• Year 1 and Year 2 data show that there is very strong scale reliability on all measures.
• Year 1 data showed moderate to very strong program effects in the majority of programs.  Year 2 data showed moderate program effects, but also showed similar movement in the desired direction among comparison group youth.
• Two of the four projects with longitudinal data showed significant pre-post movement on most or all of the short-term predictors as compared to the comparison groups, which showed no change.  The other two projects did not show short-term change on the short-term measures. This is interesting to us for several reasons.  With respect to our evaluation model, we would expect those programs which show change in short term predictors, if they really good predictors, to also show change in short term predictors, if they really are good predictors, to also show change in the longer term – on the behavioral data.

Highlights

Virginia has been selected to be one of only four states to present their evaluation study at the national abstinence evaluator’s workshop in July 2000.  Selection criteria included strength of design, positive progression of the study, and availability of data.

The Association of Maternal and Child Health invited Virginia to participate as a panelist for a teleconference on abstinence program evaluation.

Most recently, we presented at both the American Public Health Association and the National Organization of Adolescent Pregnancy Parenting and Prevention conferences.

Table 1:  The Virginia Abstinence Education Initiative
Survey Administration and Longitudinal Tracking Timetable

Table 2:  Key Predictors of Behavior Intentions (1^st Year Data)