University of Guelph Evaluates Online Outcomes Assessment Tool
The University of Guelph was one of the first institutions to define the skills students would learn in their programs. This evolved into a list of five institutional learning outcomes that were approved in 2012. As a next step, researchers worked in collaboration with Desire2Learn (D2L), a provider of learning management systems, to align and refine course- and program-level outcomes and develop a process to assess them.
A new study published by the Higher Education Quality Council of Ontario, Integrating Pedagogy and Technology to Measure Program and Institutional Learning Outcomes at the University of Guelph, finds that the online learning-outcomes assessment tool was able to effectively capture and assess learning outcomes at the program and institutional level.
Project description
A pilot study to test the assessment tool was conducted within two degree programs: Bachelor of Arts and Science (BAS) and Bachelor of Engineering (BEng). It used the learning-outcomes assessment tool in conjunction with the learning management system to capture students’ acquisition of skills and abilities and measure them over time.
The project’s three objectives were (1) the technical development of a learning-outcomes assessment system that met both disciplinary and university outcomes as well as fostered engagement and departmental participation; (2) pilot-test the system and the measurement of skill acquisition; and (3) conduct a longitudinal evaluation and analysis.
Findings
The first phase of the pilot took place within a first-year course over one semester in the BAS program. The data from the pilot illustrated that it was possible to measure student achievement of learning outcomes within the learning management system at the level of a learning activity within a course. The proportion of students achieving a satisfactory performance level increased as they progressed through the assignments.
The research team then attempted to scale up the tool from a single course to a whole program (BEng) incorporating the lessons they’d learned from the BAS phase of the research. The BEng program is much more prescribed at the course level and also incorporates an accreditor’s list of program and institutional learning outcomes. In this phase of the project, researchers worked on developing rubrics that mapped to graduate attributes. This allowed for assignments to be linked to overall program measures.
The authors found that having data mapped in the learning management system and tied to assessments enabled instructors to examine the quality of the assessments in relation to program learning outcomes. “Analyzing the gap between what is intended and what is achieved is crucial to continuously improving a curriculum” they say.
The results of this project demonstrate that the online learning-assessment tool is able to capture learning-outcomes achievement data from a variety of assignments. It also served to deepen faculty engagement in program assessment and pedagogy by providing instructors with data, providing an opportunity to discuss the data, and facilitating practical solutions and approaches that can be implemented incrementally. However, the volume of data on student achievement of outcomes raised issues related to access, security, storage and privacy. The team at Guelph is exploring ways of addressing these concerns.
The authors of Integrating Pedagogy and Technology to Measure Program and Institutional Learning Outcomes at the University of Guelph are Serge Desmarais, Michelle Fach and Dale Lackeyram.
The University of Guelph is among the Ontario postsecondary institutions participating in HEQCO’s Learning Outcomes Assessment Consortium. Each is developing and piloting assessment tools and techniques ranging from ePortfolios to analytic rubrics that will be scalable to the institution level in the future.