[ Home ] [ Up ]

Return to Table of Contents for More Strategies

Many educational programs evaluate their success by using a pre- and posttest.  The pretest measures abilities or knowledge before information or skills are presented in a program.  After the program, knowledge and skills are again measured to determine if learning outcomes have been met.  Sometimes the pre- and posttest offer an excellent measurement of success.  However, there are occasions when this traditional measurement is less helpful, especially when it involves self-reporting. During the pretest, participants—for a variety of reasons, including low self-awareness and ego protection—may inflate their assessments of knowledge, skills, or personal qualities. The subsequent posttest, then, may reflect a lower self-assessment after the educational program.

By way of example, we experienced this problem with students in our freshman seminar program which uses the On Course book by Skip Downing. In the book’s self-assessment, students are asked to rate themselves on eight inner qualities of success: personal responsibility, self-motivation, self-management, interdependence, self-awareness, lifelong learning, emotional intelligence, and self-esteem.  We found that after students have worked through the book and developed an increased understanding of themselves and these qualities, they sometimes give themselves lower self-ratings when they take the book’s posttest.  Some astute students recognize that they were unrealistic in the pre-test ratings.  Other students, however, complain that they spent a semester in the class and actually got worse.

In 1980, G.S. Howard identified a problem with using a pre- and posttest to evaluate an intervention and called it a “response shift effect.”  (Lamb & Tschillard; Pratt, et. al.)  He noted that this response shift occurs when participants think they already know the material but really do not.  When using a pre- and posttest, researchers assume participants’ frame of reference or evaluation standard will not change over the assessment period.  However, students using the On Course book often experience a dramatic change of reference as their self-awareness changes, and then, ironically, their improvement is not always reflected by their self-assessment scores. 

One possible solution to a response shift effect is to administer a retrospective pretest. A retrospective pretest administers both the pre- and posttest only after the intervention.  This approach asks participants after an intervention to assess their skills, knowledge or personal qualities both after the intervention and, retrospectively, before the intervention. In other words, how would you rate yourself now…and how would you rate how you were back when the course began?  A retrospective pretest was first proposed by Campbell and Stanley in 1963 as a quasi-experimental design to be used when it is ineffective or inefficient to use a more traditional experimental design (Campbell & Stanley).

Several studies have been conducted on the use of retrospective pretests in situations when participants might under- or overestimate their knowledge, skills or qualities when taking a traditional pretest. (Sibthorp, et. al.)  Some studies have also conducted a traditional pre- and posttest along with a retrospective pretest and then compared the results.  Results indicate that participants both under- and overestimated knowledge on the traditional pre- and posttests when compared to a retrospective pretest.  (Lamb & Tschillard; Pratt, et. al.)  Our experience is that traditional students in our freshman seminar class often overestimate their abilities at the beginning of the course.  For example, one of the statements from the On Course pretest is “When I get off course from my goals and dreams, I realize it right away.” Students often give themselves a fairly high rating at the beginning of the semester even though for many new college students their only goal is a vague one about getting a college degree so they can earn a lot of money.  If this statement were part of a retrospective pretest at the end of the semester, students would likely recognize that they now understand and have some definite goals that simply didn’t exist at the beginning of the semester. 

Retrospective pretests can also be used when there is no time or means to conduct a traditional pre/posttest, or when participants may not like the idea of being considered students or research subjects.  These tests are simple to administer and can be changed easily as programs adapt to participant needs.  Retrospective pretests are also valuable when training involves vocabulary the participants may not be ready to understand at the beginning.  Finally, the data provided by retrospective pretests, along with other supporting data, can be used to more successfully evaluate the effectiveness of an intervention or program.

Retrospective pretests are not beneficial for all situations.  If used for a college course, students must think back several months and try to recall their level of knowledge at the beginning of the course, which is also subject to bias.  Retrospective pretests can also be perceived as less rigorous than more traditional assessments and results may not be accepted as a valid reflection of improvement since evaluation is based on self-report. 

Despite these potential drawbacks, we found the use of a retrospective self-assessment to be helpful in assessing our course objectives for our Foundations for Success freshman program that uses the On Course text. Some of our course objectives relate directly to concepts in the On Course book.  (See chart below)  Students were given the opportunity to rate where they thought they placed at that moment on a ten-point scale.  Then they rated where they believe they were at the beginning of the semester.  Students had an opportunity to indicate their belief that the course achieved what it was designed to achieve, and also had the opportunity to recognize their personal improvement.  Using a retrospective pretest, our findings have shown that on a 10 point scale, students have rated themselves, on average, 4 points higher than at the beginning of the semester.

Some faculty took this concept even further.  Students also had taken the assessment designed by the author of the book at the beginning of the semester and again at the end of the semester.  Those students used both the book’s pre- and post assessment and the retrospective pretest to reflect on their perceived growth as a college student and an adult. They were challenged to analyze why their self-measures on the pre- and posttest differed and how those scores differed from the scores on the retrospective pretest.  In their written reflections, the students often reached the conclusion that their early scores at the beginning of the semester indicated they really hadn’t been honest with themselves or didn’t truly understand the concepts. One student said, “I don’t remember what I was thinking when I did the first test.  I must have just been worried about getting it done because it really wasn’t very accurate about me.”   Students reached their own very satisfactory conclusion about their growth and were far less often disappointed in their end-of-term assessment from the book.  Faculty did not need to deal with student disappointment or intervene in this last opportunity of the semester to gain greater insight into themselves. This approach provided both faculty and administration a better opportunity to evaluate perceived student success.   

Retrospective pretests are easy to design for many learning situations and can provide insight for the faculty as well as further demonstrate students’ assessment of their own skills.  By asking the students’ opinions, the students may feel they are more involved in the learning process. This could prove a very empowering experience by students’ furthering their understanding that they really are ultimately responsible for their own learning. 

REFERENCES

Campbell, D.T., Stanley, J. C. (1963). Experimental and Quasi-experimental Designs for Research. Chicago : Rand McNally.

Lamb, T. (2005). "The retrospective pretest: An imperfect but useful tool." The Evaluation Exchange, 11(2). Retrieved October, 19, 2005 , from http://www.gse.harvard.edu/hfrp/eval/issue30/spotlight.html [Note:  The url provided returned invalid results on March 1, 2010 . Visit homepage.]

http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/ Evaluation Methodology, Volume XI, No. 2, Summer 2005

Lamb, T. A., & Tschillard, R. (2005). "Evaluating learning in professional development workshops: Using the retrospective pretest." Journal of Research in Professional Learning, Retrieved March 6, 2009 http://www.nsdc.org/library/publications/research/lamb.pdf.

Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000). "Measuring Program Outcomes: Using Retrospective Pretest Methodology." American Journal of Evaluation, 21(3), 341.

Sibthorp, J., Paisley , K., Gookin, J., & Ward, P. (2007). "Addressing Response-shift Bias: Retrospective Pretests in Recreation Research and Evaluation." Journal of Leisure Research, 39(2), 295-315.

by Barbara Kirkwood, Associate Director, Group Academic Support, Indiana University Purdue University-Forth Wayne , IN

* * * * *