Recently, the Joint Committee for the Management the Agreement (JCMA) met with Vice-Provost, Teaching, Learning and Student Experience Patti McDougall and Teaching and Learning Centre Director Nancy Turner to discuss concerns raised by the Ryerson arbitration decision and a recent report from the Ontario Confederation of University Faculty Associations (OCUFA). JCMA also heard about their experiences with pilots of the new “Student Learning Experience Questionnaire” (SLEQ), a name reflective of a change from the language of “teaching evaluation” to “learning experience.”
We have reported to you in recent months regarding national events concerning student evaluation of teaching. Most recently, in an e-Letter, we told you about the OCUFA report stemming from an investigation of complaints of misuse and inappropriate interpretations of scores on student evaluations of teaching. The report found that even responses to questions about specific features – the length of time in returning assignments, for example – are affected by respondents’ broader assessment of an instructor, and thereby impacted by stereotypical views, views that are often unconsciously held. An earlier e-Letter asked you to comment on the Ryerson decision where the arbitrator ordered that student-based evaluations of teaching not be used to measure teaching effectiveness in tenure and promotion cases, effective immediately, until the parties could negotiate appropriate language in their collective agreement. A basis for the arbitration ruling is the extensive literature about student evaluations of teaching that show results from these instruments are influenced by instructors’ gender, race, ethnicity, age, physical attractiveness, class size, time of day, etc.
Both guests knew about the Ryerson decision and both were aware of, and agreed with, many of the concerns about the biases inherent in student evaluations expressed by the OCUFA report. They stated that SLEQ asks students to respond to questions that focus on the course rather than the instructor, such as: “The course provided me with a deeper understanding of the subject matter,” “Course projects, assignments, tests, and/or exams improved my understanding of the course material.” The instructor’s unit may add other questions, and so may the Instructor.
They also stated that this is not just a new form and a new system, but also raises the issue how the feedback of students is used. Decision makers will need to be educated appropriately to evaluate these forms as a single element within a case file. Using only student questionnaires is no longer considered a best practice. Additionally, they stated, this will require decision makers to understand sample sizes, and frequency distributions rather than simple averages, as well as how to look out for potential biases.
SLEQ will not be imposed on any unit. Ultimately, it is up to units to determine what instrument they wish to use to collect student feedback. Instructors will be encouraged to use class time to complete the questionnaires. Students will be allowed to opt out if classes are so small that their comments might easily be attributed to them. As well, faculty will be allowed to request that evaluations containing inappropriate remarks be removed in their entirety.
USFA is following this issue closely and will continue to report on new developments. We recognize that the stakes can be high for faculty members’ careers with the continuing use of student evaluations in tenure and promotion decisions. In the meantime, faculty members in individual units should remember that no anonymous material should be introduced or considered, except for student course evaluations, which must be properly validated instruments of performance evaluation. Whether or not they choose to adopt SLEQ, academic units should consider, and articulate how they will address the problem of bias in student responses – that is, the conscious or unconscious views about instructor’s gender, race, ethnicity, age, physical attractiveness, class size, time of day, etc. As well, since SLEQ results will not be released if the response rate or class size is below a certain threshold, units should also consider how issues of small classes and low response rates will be addressed, whatever instrument they choose.