Online Student Evaluations: Filling in the Blanks
—By Steve Newman
Steve Newman, Associate Professor of English
Since the Fall of 2009, the Faculty Senate Student Feedback Form Committee has been piloting an online version of student evaluations. The advantages of switching to online evaluations are clear enough. It would lead to significant savings, not just of trees and money but also of the tremendous labor of those who have to assemble, transport, and process all those packets of paper. It would also be nice not to have to devote precious class time to filling out the forms. I would be thrilled, then, for Temple to make the switch . . . but only if the rate and quality of the responses for the electronic forms (e-SFFs) can be raised to a level close to those of the paper forms. There is too much at stake—in improving our teaching, and in making well-informed decisions about promotion, tenure, renewal, and merit—to adopt a system that lacks validity. Some faculty members dismiss student evaluations as a mere popularity contest and so don’t see what the fuss is about in going to electronic forms. I strongly disagree. Although I’ve read my share of silly evaluations, I have also read many that have pointed intelligently to areas where I could improve, sometimes confirming what I suspected and other times identifying shortcomings I was blind to. (And, of course, the praise is always nice when it comes.)
The numbers suggest how far the e-SFFs have to go. During the pilot, the response rates for the paper forms have ranged from 72 to 80%; for the e-SFFs, from 26 to 44%. I have talked to many colleagues who use statistical analysis in their work, and while positions certainly vary about the threshold for validity, I have talked with no one who thinks 44% is sufficient, let alone 26%. This is especially true in a case where those who have not responded do not seem to have done so randomly. It seems very likely that those who opt to fill out the online evaluations are more likely motivated by strong likes or dislikes of the class—many faculty I’ve talked to are particularly troubled by the latter scenario. The SFF Committee has begun investigating this: Comparing over 200 sections, they have found no statistically significant difference in the numerical evaluations for paper and electronic SFFs for identical courses taught by the same faculty members. So perhaps concerns about response rates need to be moderated, but the committee itself still views the e-SFF rates asa problem since they continue to seek ways to raise them. A final piece of data: The response rates to the non-numerical/discursive questions were lower for the electronic SFFs across the curriculum: 62-34% in lower-level courses, 53-28% in upper-level courses, and 71%-34% in graduate/ professional courses.
There are problems beyond what’s revealed by the aggregate numbers. In response to a query I sent out to the TAUP listserv, I received many troubling reports. One colleague informed me that there were wide variations in e-SFFs from sections of the same course, with the one under 33% much more negative: “I think it's pretty unlikely that I actually taught one section so much better than the other. Much more likely is that, especially with small sections, the low response rates associated with the online format can easily produce unrepresentative responses.” Another colleague received only 4 out of 16 responses: “The written comments are especially important to me; the final question on the SFFs is one of the only places at Temple University in which sexual identity and orientation is noted in terms of inclusion. A lot of my classes link to that important topic.” A junior colleague received only 3 of 24 e-SFFs, which rendered the class useless as a way to improve her teaching, not to mention for promotion, tenure, and merit; another who received no written comments in her graduate or undergraduate courses and not surprisingly expressed grave disappointment.
These responses illustrate a few additional points of concern. The lower response rates make it more likely that smaller classes will fall below the threshold of 8 required for the forms to be processed, a frustrating outcome that leads faculty to wonder why they should engage in this exercise at all. More troubling still is the decline in responses to the discursive question—both in the number and quality of the responses. This problem has been compounded by the difficulty in folding in the additional questions many departments and programs have developed, and by a glitch in the program that cut off many students’ responses. I am assured by Sally Frazee of the Measurement and Research Center and a member of the SFF Committee that the latter has been fixed in time for this term’s evaluations. I hypothesize, though, that even with this correction and even if students become more accustomed to the electronic forms, they will continue to produce fewer and less valuable discursive responses. Why? The nature of the electronic format makes it more likely that they will devote less time and focus to these evaluations. It is true that students could simply space out in class when they should be filling out the paper forms or interrupt themselves by checking Facebook on their phones or laptops; conversely, it is possible that they may take advantage of the increased time offered by the online medium to write fuller responses. But my own experience
and other reports from colleagues suggest that students are more likely to be distracted and less engaged when filling these out in dorm rooms, apartments, and computer labs. This is particularly troubling because the written responses are, to my mind, the more valuable part of the evaluations. As Joseph DuCette, a professor in the College of Education and a member of the SFF Committee puts it, evaluations serve both a “formative” and “summative” function. A decline in the discursive responses saps both functions, particularly the formative. Perhaps that is a price we have to pay for the savings. I’m not sure it’s worth it.
Prof. DuCette also made it clear that the SFF Committee is aware of many of these problems and is looking to promising models for increasing response rates without simply coercing the students (say, by not releasing their grades unless they fill them out). Among these models: A student would gain access to data from the class as a whole only if she filled out the evaluation and a certain percentage of her classmates did as well. In my exchanges with the members of the committee, they have shared their data generously and responded thoughtfully to my questions. Yet they apparently have not yet set a clear and defensible benchmark that would justify a university-wide switch to e-SFFs; neither have they taken up the question of the quality of the openended responses, though they plan to. We must encourage them to address these matters effectively and to continue communicating with us, our students, and other members of the university community. If this is a pilot program rather than a fait accompli, there must be clear criteria that would moveus to walk away from the e-SFF, however disappointing that would be. We must also use this debate on the SFFs to build on the valuable discussions in our departments and colleges and the Teaching and Learning Center on teaching; however important student evaluations are, they must be only one dimension in evaluating and improving teaching. If we do not do all of this, we risk missing the forest for the trees.