Research Summary:
Student Engagement Surveys Forge Pathways to Quality Improvement But Can Mislead when Used for Competitive Rankings
Student engagement surveys are helping colleges and universities, and the postsecondary system overall, be more responsive to the learning needs of students. Designed in part to measure the time, effort and interest student invest in their education, the dominant surveys have been developed in the United States – key among them the university-focused National Survey of Student Engagement (NSSE) and the Community College Survey of Student Engagement (CCSSE).
But are these surveys reliable in the Ontario context? Can they predict learning outcomes, guide and inform institutional planning, and be used by the government for institutional or system accountability? And perhaps most important for institutions seeking a competitive advantage: should engagement survey results be used for rankings and direct comparative purposes?
NSSE is administered on a regular basis to first and fourth-year Ontario undergraduate university students. CCSSE has not been as widely used by Ontario colleges but was recently tested for validity at Humber College. Currently, colleges utilize provincially mandated key performance indicators (KPIs) to measure the satisfaction of students, graduates and employers as well as each college’s graduation and employment rates. In a two-year pilot with several Ontario colleges, a series of engagement questions has been added to the existing KPI survey.
Project descriptions
The NSSE National Data Project Report and the @ Issue paper Student Engagement as a Quality Measure in the Ontario Postsecondary System from the Higher Education Quality Council of Ontario (HEQCO), indicate that while engagement surveys are important resources that can create pathways to quality improvement, they can lead to misleading conclusions when used for competitive ranking purposes.
The NSSE National Data Project, commissioned by HEQCO, is an ambitious, cross-Canada analysis of NSSE student responses and student records data from 44 universities. Among its goals is helping institutions identify student engagement strengths and weaknesses independent of fixed characteristics such as institutional size and mix of students. With the data, institutions can more effectively identify best results and practices that lead to more engaged students. The study is referenced in the @ Issue paper on student engagement, which synthesizes findings from this and three other studies on student engagement published by HEQCO over the last year.
Findings
Variations in engagement results among colleges and universities are associated with many factors including institutional characteristics, student composition and program mix. In fact, the three account for much of the variation in university engagement scores. As a result, postsecondary institutions should not use the results for competitive positioning, but focus An agency of the Government of Ontario instead on individual institutional findings to identify areas for quality improvement based on their respective missions and goals.
Both the NSSE and CCSSE surveys are generally valid and reliable tools in the Canadian context and student engagement measures may help predict learning outcomes, according to the HEQCO research. Continued participation in engagement-related surveys by Ontario’s colleges and universities and their continued use by the Province of Ontario as part of the system’s accountability framework are recommended.
Further, longitudinal surveys and data linkages between survey responses and administrative data can provide a deeper understanding of student engagement, which can then be utilized to design or improve engagement strategies. Student engagement survey data could be improved through larger sample sizes and increased response rates.
While a number of postsecondary institutions have used engagement survey results to help design improvement interventions, there is insufficient information sharing among institutions. There should be more inter-institutional collaboration and learning in terms of successful intervention practices, useful assessment tools and appropriate research methodologies. Government should also promote and support information sharing between institutions.
In general, with continued participation, assessment and retesting, measuring educational quality in the Ontario postsecondary education sector can help Ontario colleges and universities
identify effective educational practices as well as areas for improvement that will keep students in school, improve their academic performance and increase graduation rates.
Authors of the NSSE National Data Project Report are Chris Conway, director of the Office of Institutional Research and Planning at Queen’s University; Huizi Zhao, HEQCO research analyst; and Sara Montgomery, research and planning analyst with the Office of Institutional Research and Planning at Queen’s University.
The @ Issue review of current research on student engagement was written by HEQCO research analyst Huizi Zhao.