Standardized international learning outcomes assessment possible but needs refining
The Organisation for Economic Co-operation and Development (OECD) Assessment of Higher Education Learning Outcomes (AHELO) feasibility study was successful in building international relationships, furthering the discussion on learning outcomes and exploring the potential for a common worldwide student assessment. But as an instrument to produce reliable, valuable and useful data for Ontario, the instrument needs refinement and improvement, according to a new report from the Higher Education Quality Council of Ontario (HEQCO).
Project description
In 2011, Ontario joined the AHELO feasibility study to determine if standardized tests could be used across countries to measure what students know and are able to do upon completing a bachelor’s-level program of study. HEQCO led the provincial project on behalf of the Ministry of Training, Colleges and Universities and in cooperation with the Council of Ministers of Education, Canada.
The feasibility study comprised three tests: one for generic skills and two for discipline-specific skills in economics and civil engineering. Ontario participated in the civil engineering strand, with nine out of ten Ontario universities with civil engineering programs involved in the study, representing almost 450 students. The participating universities were: Carleton, McMaster, Ottawa, Queen’s, Ryerson, Toronto, Waterloo, Western and Windsor. Students were tested between February and June 2012. AHELO: The Ontario Experience details Ontario’s participation in the international feasibility study.
Findings
One of the primary goals of the AHELO project was to determine if it was possible to implement a standard online assessment, in a common way, to students around the world. This project suggested that it might be possible. Experts and faculty agreed on shared learning outcomes and assessment questions, and the project management and execution followed a common protocol across the globe.
The AHELO feasibility study was not intended to be a ranking exercise. So, from the outset, it was clear that no comparisons between institutions or jurisdictions would be made. It was also understood that the data collected were from an unrepresentative sample of jurisdictions and institutions. In addition, the assessments made in this feasibility study were not sensitive enough to provide institution-level information by specific competency area nor could they provide reliable feedback for individual students.
The feasibility study revealed international interest in large-scale learning outcomes assessments, but the continuing need to refine and improve the assessment instrument if meaningful and valuable conclusions are to be drawn from the data.
Authors of AHELO: The Ontario Experience are Mary Catharine Lennon and Linda Jonker, HEQCO. Previous reports on AHELO include the Feasibility Study Report, Volume 1: Design and Implementation , Volume 2: Data Analysis and National Experiences and Volume 3: Further Insights .